[openstack-dev] [all][api] POST /api-wg/news

[Edward Leafe]

Greetings OpenStack community,

Today's meeting was even shorter and sweeter than last week's, as only edleafe 
was present, due to cdent being on PTO and elmiko being on the road. Prior to 
the meeting, though, we had decided that the guideline change for raising the 
minimum microversion was ready for freeze, and so it was indeed frozen. Other 
than that, there really is nothing else to report.

# Newly Published Guidelines

None this week

# API Guidelines Proposed for Freeze

Guidelines that are ready for wider review by the whole community.

* Microversions: add next_min_version field in version body
  https://review.openstack.org/#/c/446138/

# Guidelines Currently Under Review [3]

* A (shrinking) suite of several documents about doing version discovery
  Start at https://review.openstack.org/#/c/459405/

* Fix html_last_updated_fmt for Python3
  https://review.openstack.org/475219

* WIP: microversion architecture archival doc (very early; not yet ready for 
review)
  https://review.openstack.org/444892

# Highlighting your API impacting issues

If you seek further review and insight from the API WG, please address your 
concerns in an email to the OpenStack developer mailing list[1] with the tag 
"[api]" in the subject. In your email, you should include any relevant reviews, 
links, and comments to help guide the discussion of the specific challenge you 
are facing.

To learn more about the API WG mission and the work we do, see OpenStack API 
Working Group [2].

Thanks for reading and see you next week!

# References

[1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[2] http://specs.openstack.org/openstack/api-wg/
[3] https://review.openstack.org/#/q/status:open+project:openstack/api-wg,n,z

Meeting Agenda
https://wiki.openstack.org/wiki/Meetings/API-WG#Agenda
Past Meeting Records
http://eavesdrop.openstack.org/meetings/api_wg/
Open Bugs
https://bugs.launchpad.net/openstack-api-wg

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Trying to understand the proposed direction

[Edward Leafe]

On Jun 20, 2017, at 8:38 AM, Jay Pipes  wrote:
> 
>>> The example I posted used 3 resource providers. 2 compute nodes with no 
>>> local disk and a shared storage pool.
>> Now I’m even more confused. In the straw man example 
>> (https://review.openstack.org/#/c/471927/ 
>> ) 
>> >  
>> >
>>  I see only one variable ($COMPUTE_NODE_UUID) referencing a compute node in 
>> the response.
> 
> I'm referring to the example I put in this email threads in 
> paste.openstack.org  with numbers showing 1600 
> bytes for 3 resource providers:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118593.html 
> 


And I’m referring to the comment I made on the spec back on June 13 that was 
never corrected/clarified. I’m glad you gave an example yesterday after I 
expressed my confusion; that was the whole purpose of starting this thread. 
Things may be clear to you, but they have confused me and others. We can’t help 
if we don’t understand.


-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Trying to understand the proposed direction

[Edward Leafe]

On Jun 20, 2017, at 6:54 AM, Jay Pipes  wrote:
> 
>>> It was the "per compute host" that I objected to.
>> I guess it would have helped to see an example of the data returned for 
>> multiple compute nodes. The straw man example was for a single compute node 
>> with SR-IOV, NUMA and shared storage. There was no indication how multiple 
>> hosts meeting the requested resources would be returned.
> 
> The example I posted used 3 resource providers. 2 compute nodes with no local 
> disk and a shared storage pool.


Now I’m even more confused. In the straw man example 
(https://review.openstack.org/#/c/471927/ 
) 

 I see only one variable ($COMPUTE_NODE_UUID) referencing a compute node in the 
response.

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Trying to understand the proposed direction

[Edward Leafe]

On Jun 19, 2017, at 5:27 PM, Jay Pipes  wrote:
> 
>> It was from the straw man example. Replacing the $FOO_UUID with UUIDs, and 
>> then stripping out all whitespace resulted in about 1500 bytes. Your 
>> example, with whitespace included, is 1600 bytes.
> 
> It was the "per compute host" that I objected to.

I guess it would have helped to see an example of the data returned for 
multiple compute nodes. The straw man example was for a single compute node 
with SR-IOV, NUMA and shared storage. There was no indication how multiple 
hosts meeting the requested resources would be returned.

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Trying to understand the proposed direction

[Edward Leafe]

On Jun 19, 2017, at 1:34 PM, Jay Pipes  wrote:
> 
>> OK, thanks for clarifying that. When we discussed returning 1.5K per compute 
>> host instead of a couple of hundred bytes, there was discussion that paging 
>> would be necessary.
> 
> Not sure where you're getting the whole 1.5K per compute host thing from.

It was from the straw man example. Replacing the $FOO_UUID with UUIDs, and then 
stripping out all whitespace resulted in about 1500 bytes. Your example, with 
whitespace included, is 1600 bytes. 

> Here's a paste with the before and after of what we're talking about:
> 
> http://paste.openstack.org/show/613129/ 
> 
> 
> Note that I'm using a situation with shared storage and two compute nodes 
> providing VCPU and MEMORY. In the current situation, the shared storage 
> provider isn't returned, as you know.
> 
> The before is 231 bytes. The after (again, with three providers, not 1) is 
> 1651 bytes.

So in the basic non-shared, non-nested case, if there are, let’s say, 200 
compute nodes that can satisfy the request, will there be 1 
“allocation_requests” key returned, with 200 “allocations” sub-keys? And one 
“provider_summaries” key, with 200 sub-keys on the compute node UUID?

> gzipping the after contents results in 358 bytes.
> 
> So, honestly I'm not concerned.

Ok, just wanted to be clear.

>> OK, that’s informative, too. Is there anything decided on how much host info 
>> will be in the response from placement, and how much will be in HostState? 
>> Or how the reporting of resources by the compute nodes will have to change 
>> to feed this information to placement? Or how the two sources of information 
>> will be combined so that the filters and weighers can process it? Or is that 
>> still to be worked out?
> 
> I'm currtently working on a patch that integrates the REST API into the 
> scheduler.
> 
> The merging of data will essentially start with the resource amounts that the 
> host state objects contain (stuff like total_usable_ram etc) with the 
> accurate data from the provider_summaries section.


So in the near-term, we will be using provider_summaries to update the 
corresponding HostState objects with those values. Is the long-term plan to 
have most of the HostState information moved to placement?


-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Trying to understand the proposed direction

[Edward Leafe]

On Jun 19, 2017, at 9:17 AM, Jay Pipes  wrote:

As Matt pointed out, I mis-wrote when I said “current flow”. I meant “current 
agreed-to design flow”. So no need to rehash that.

>> * Placement returns a number of these data structures as JSON blobs. Due to 
>> the size of the data, a page size will have to be determined, and placement 
>> will have to either maintain that list of structured datafor subsequent 
>> requests, or re-run the query and only calculate the data structures for the 
>> hosts that fit in the requested page.
> 
> "of these data structures as JSON blobs" is kind of redundant... all our REST 
> APIs return data structures as JSON blobs.

Well, I was trying to be specific. I didn’t mean to imply that this was a 
radical departure or anything.

> While we discussed the fact that there may be a lot of entries, we did not 
> say we'd immediately support a paging mechanism.

OK, thanks for clarifying that. When we discussed returning 1.5K per compute 
host instead of a couple of hundred bytes, there was discussion that paging 
would be necessary.

>> * Scheduler continues to request the paged results until it has them all.
> 
> See above. Was discussed briefly as a concern but not work to do for first 
> patches.
> 
>> * Scheduler then runs this data through the filters and weighers. No 
>> HostState objects are required, as the data structures will contain all the 
>> information that scheduler will need.
> 
> No, this isn't correct. The scheduler will have *some* of the information it 
> requires for weighing from the returned data from the GET 
> /allocation_candidates call, but not all of it.
> 
> Again, operators have insisted on keeping the flexibility currently in the 
> Nova scheduler to weigh/sort compute nodes by things like thermal metrics and 
> kinds of data that the Placement API will never be responsible for.
> 
> The scheduler will need to merge information from the "provider_summaries" 
> part of the HTTP response with information it has already in its HostState 
> objects (gotten from ComputeNodeList.get_all_by_uuid() and 
> AggregateMetadataList).

OK, that’s informative, too. Is there anything decided on how much host info 
will be in the response from placement, and how much will be in HostState? Or 
how the reporting of resources by the compute nodes will have to change to feed 
this information to placement? Or how the two sources of information will be 
combined so that the filters and weighers can process it? Or is that still to 
be worked out?

>> * Scheduler then selects the data structure at the top of the ranked list. 
>> Inside that structure is a dict of the allocation data that scheduler will 
>> need to claim the resources on the selected host. If the claim fails, the 
>> next data structure in the list is chosen, and repeated until a claim 
>> succeeds.
> 
> Kind of, yes. The scheduler will select a *host* that meets its needs.
> 
> There may be more than one allocation request that includes that host 
> resource provider, because of shared providers and (soon) nested providers. 
> The scheduler will choose one of these allocation requests and attempt a 
> claim of resources by simply PUT /allocations/{instance_uuid} with the 
> serialized body of that allocation request. If 202 returned, cool. If not, 
> repeat for the next allocation request.

Ah, yes, good point. A host with multiple nested providers, or with shared and 
local storage, will have to have multiple copies of the data structure returned 
to reflect those permutations. 

>> * Scheduler then creates a list of N of these data structures, with the 
>> first being the data for the selected host, and the the rest being data 
>> structures representing alternates consisting of the next hosts in the 
>> ranked list that are in the same cell as the selected host.
> 
> Yes, this is the proposed solution for allowing retries within a cell.

OK.

>> * Scheduler returns that list to conductor.
>> * Conductor determines the cell of the selected host, and sends that list to 
>> the target cell.
>> * Target cell tries to build the instance on the selected host. If it fails, 
>> it uses the allocation data in the data structure to unclaim the resources 
>> for the selected host, and tries to claim the resources for the next host in 
>> the list using its allocation data. It then tries to build the instance on 
>> the next host in the list of alternates. Only when all alternates fail does 
>> the build request fail.
> 
> I'll let Dan discuss this last part.


Well, that’s not substantially different than the original plan, so no 
additional explanation is required.

One other thing: since this new functionality is exposed via a new API call, is 
the existing method of filtering RPs by passing in resources going to be 
deprecated? And the code for adding filtering by traits to that also no longer 
useful?


-- Ed Leafe

[openstack-dev] [nova][scheduler][placement] Trying to understand the proposed direction

[Edward Leafe]

There is a lot going on lately in placement-land, and some of the changes being 
proposed are complex enough that it is difficult to understand what the final 
result is supposed to look like. I have documented my understanding of the 
current way that the placement/scheduler interaction works, and also what I 
understand if how it will work when the proposed changes are all implemented. I 
don’t know how close that understanding is to what the design is, so I’m hoping 
that this will serve as a starting point for clarifying things, so that 
everyone involved in these efforts has a clear view of the target we are aiming 
for. So please reply to this thread with any corrections or additions, so that 
all can see.

I do realize that some of this is to be done in Pike, and the rest in Queens, 
but that timetable is not relevant to the overall understanding of the design.

-- Ed Leafe

Current flow:
* Scheduler gets a req spec from conductor, containing resource requirements
* Scheduler sends those requirements to placement
* Placement runs a query to determine the root RPs that can satisfy those 
requirements
* Placement returns a list of the UUIDs for those root providers to scheduler
* Scheduler uses those UUIDs to create HostState objects for each
* Scheduler runs those HostState objects through filters to remove those that 
don't meet requirements not selected for by placement
* Scheduler runs the remaining HostState objects through weighers to order them 
in terms of best fit.
* Scheduler takes the host at the top of that ranked list, and tries to claim 
the resources in placement. If that fails, there is a race, so that HostState 
is discarded, and the next is selected. This is repeated until the claim 
succeeds.
* Scheduler then creates a list of N UUIDs, with the first being the selected 
host, and the the rest being alternates consisting of the next hosts in the 
ranked list that are in the same cell as the selected host.
* Scheduler returns that list to conductor.
* Conductor determines the cell of the selected host, and sends that list to 
the target cell.
* Target cell tries to build the instance on the selected host. If it fails, it 
unclaims the resources for the selected host, and tries to claim the resources 
for the next host in the list. It then tries to build the instance on the next 
host in the list of alternates. Only when all alternates fail does the build 
request fail.

Proposed flow:
* Scheduler gets a req spec from conductor, containing resource requirements
* Scheduler sends those requirements to placement
* Placement runs a query to determine the root RPs that can satisfy those 
requirements
* Placement then constructs a data structure for each root provider as 
documented in the spec. [0]
* Placement returns a number of these data structures as JSON blobs. Due to the 
size of the data, a page size will have to be determined, and placement will 
have to either maintain that list of structured datafor subsequent requests, or 
re-run the query and only calculate the data structures for the hosts that fit 
in the requested page.
* Scheduler continues to request the paged results until it has them all.
* Scheduler then runs this data through the filters and weighers. No HostState 
objects are required, as the data structures will contain all the information 
that scheduler will need.
* Scheduler then selects the data structure at the top of the ranked list. 
Inside that structure is a dict of the allocation data that scheduler will need 
to claim the resources on the selected host. If the claim fails, the next data 
structure in the list is chosen, and repeated until a claim succeeds.
* Scheduler then creates a list of N of these data structures, with the first 
being the data for the selected host, and the the rest being data structures 
representing alternates consisting of the next hosts in the ranked list that 
are in the same cell as the selected host.
* Scheduler returns that list to conductor.
* Conductor determines the cell of the selected host, and sends that list to 
the target cell.
* Target cell tries to build the instance on the selected host. If it fails, it 
uses the allocation data in the data structure to unclaim the resources for the 
selected host, and tries to claim the resources for the next host in the list 
using its allocation data. It then tries to build the instance on the next host 
in the list of alternates. Only when all alternates fail does the build request 
fail.


[0] https://review.openstack.org/#/c/471927/





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Allocating Complex Resources

[Edward Leafe]

On Jun 12, 2017, at 10:20 AM, Jay Pipes  wrote:

>> The RP uuid is part of the provider: the compute node's uuid, and (after 
>> https://review.openstack.org/#/c/469147/ merges) the PCI device's uuid. So 
>> in the code that passes the PCI device information to the scheduler, we 
>> could add that new uuid field, and then the scheduler would have the 
>> information to a) select the best fit and then b) claim it with the specific 
>> uuid. Same for all the other nested/shared devices.
> 
> How would the scheduler know that a particular SRIOV PF resource provider 
> UUID is on a particular compute node unless the placement API returns 
> information indicating that SRIOV PF is a child of a particular compute node 
> resource provider?


Because PCI devices are per compute node. The HostState object populates itself 
from the compute node here:

https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L224-L225
 


If we add the UUID information to the PCI device, as the above-mentioned patch 
proposes, when the scheduler selects a particular compute node that has the 
device, it uses the PCI device’s UUID. I thought that having that information 
in the scheduler was what that patch was all about.

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][api] POST /api-wg/news

[Edward Leafe]

On Jun 8, 2017, at 11:59 AM, Chris Dent  wrote:
> 
> The naming issue related to an optional field we want to add to the 
> microversion discovery document. Some projects wish to be able to signal that 
> they are intending to raise the minimum microversion at a point in the 
> future. The name for the next minimum version is fairly clear: 
> "next_min_version". What's less clear is the name which can be used for the 
> field that states the earliest date at which this will happen. This cannot be 
> a definitive date because different deployments will release the new code at 
> different times. We can only say "it will be no earlier than this time".
> 
> Naming this field has proven difficult. The original was "not_before", but 
> that has no association with "min_version" so is potentially confusing. 
> However, people who know how to parse the doc will know what it means so it 
> may not matter. As always, naming is hard, so we seek input from the 
> community to help us find a suitable name. This is something we don't want to 
> ever have to change, so it needs to be correct from the start. Candidates 
> include:
> 
> * not_before
> * not_raise_min_before
> * min_raise_not_before
> * earliest_min_raise_date
> * min_version_eol_date
> * next_min_version_effective_date
> 
> If you have an opinion on any of these, or a better suggestion please let us 
> know, either on the review at  >, or in response to this message.


Even better: please respond on the SurveyMonkey page to rank your preferences!

https://www.surveymonkey.com/r/L7XWNG5 



-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Allocating Complex Resources

[Edward Leafe]

Sorry for the top-post, but it seems that nobody has responded to this, and 
there are a lot of important questions that need answers. So I’m simply 
re-posting this so that we don’t get too ahead of ourselves, by planning 
implementations before we fully understand the problem and the implications of 
any proposed solution.


-- Ed Leafe


> On Jun 6, 2017, at 9:56 AM, Chris Dent  wrote:
> 
> On Mon, 5 Jun 2017, Ed Leafe wrote:
> 
>> One proposal is to essentially use the same logic in placement
>> that was used to include that host in those matching the
>> requirements. In other words, when it tries to allocate the amount
>> of disk, it would determine that that host is in a shared storage
>> aggregate, and be smart enough to allocate against that provider.
>> This was referred to in our discussion as "Plan A".
> 
> What would help for me is greater explanation of if and if so, how and
> why, "Plan A" doesn't work for nested resource providers.
> 
> We can declare that allocating for shared disk is fairly deterministic
> if we assume that any given compute node is only associated with one
> shared disk provider.
> 
> My understanding is this determinism is not the case with nested
> resource providers because there's some fairly late in the game
> choosing of which pci device or which numa cell is getting used.
> The existing resource tracking doesn't have this problem because the
> claim of those resources is made very late in the game. < Is this
> correct?
> 
> The problem comes into play when we want to claim from the scheduler
> (or conductor). Additional information is required to choose which
> child providers to use. <- Is this correct?
> 
> Plan B overcomes the information deficit by including more
> information in the response from placement (as straw-manned in the
> etherpad [1]) allowing code in the filter scheduler to make accurate
> claims. <- Is this correct?
> 
> For clarity and completeness in the discussion some questions for
> which we have explicit answers would be useful. Some of these may
> appear ignorant or obtuse and are mostly things we've been over
> before. The goal is to draw out some clear statements in the present
> day to be sure we are all talking about the same thing (or get us
> there if not) modified for what we know now, compared to what we
> knew a week or month ago.
> 
> * We already have the information the filter scheduler needs now by
>  some other means, right?  What are the reasons we don't want to
>  use that anymore?
> 
> * Part of the reason for having nested resource providers is because
>  it can allow affinity/anti-affinity below the compute node (e.g.,
>  workloads on the same host but different numa cells). If I
>  remember correctly, the modelling and tracking of this kind of
>  information in this way comes out of the time when we imagined the
>  placement service would be doing considerably more filtering than
>  is planned now. Plan B appears to be an acknowledgement of "on
>  some of this stuff, we can't actually do anything but provide you
>  some info, you need to decide". If that's the case, is the
>  topological modelling on the placement DB side of things solely a
>  convenient place to store information? If there were some other
>  way to model that topology could things currently being considered
>  for modelling as nested providers be instead simply modelled as
>  inventories of a particular class of resource?
>  (I'm not suggesting we do this, rather that the answer that says
>  why we don't want to do this is useful for understanding the
>  picture.)
> 
> * Does a claim made in the scheduler need to be complete? Is there
>  value in making a partial claim from the scheduler that consumes a
>  vcpu and some ram, and then in the resource tracker is corrected
>  to consume a specific pci device, numa cell, gpu and/or fpga?
>  Would this be better or worse than what we have now? Why?
> 
> * What is lacking in placement's representation of resource providers
>  that makes it difficult or impossible for an allocation against a
>  parent provider to be able to determine the correct child
>  providers to which to cascade some of the allocation? (And by
>  extension make the earlier scheduling decision.)
> 
> That's a start. With answers to at last some of these questions I
> think the straw man in the etherpad can be more effectively
> evaluated. As things stand right now it is a proposed solution
> without a clear problem statement. I feel like we could do with a
> more clear problem statement.
> 
> Thanks.
> 
> [1] https://etherpad.openstack.org/p/placement-allocations-straw-man
> 
> -- 
> Chris Dent  ┬──┬◡ﾉ(° -°ﾉ)   https://anticdent.org/
> freenode: cdent tw: 
> @anticdent__
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 

Re: [openstack-dev] [ironic][nova] Goodbye^W See you later

[Edward Leafe]

On Jun 8, 2017, at 7:45 AM, Jim Rollenhagen  wrote:

> I've been mostly missing for the past six weeks while looking for a new job, 
> so maybe you've forgotten me already, maybe not. I'm happy to tell you I've 
> found one that I think is a great opportunity for me. But, I'm sad to tell 
> you that it's totally outside of the OpenStack community.

Glad you found something new and interesting. I’m sure you’ll continue to do 
great things there!

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Allocating Complex Resources

[Edward Leafe]

On Jun 7, 2017, at 1:44 PM, Mooney, Sean K > wrote:

> [Mooney, Sean K] neutron will need to use nested resource providers to track
> Network backend specific consumable resources in the future also. One example 
> is
> is hardware offloaded virtual(e.g. vitio/vhost-user) interfaces which due to
> their hardware based implementation are both a finite consumable
> resources and have numa affinity and there for need to track and nested.
> 
> Another example for neutron would be bandwidth based scheduling / sla 
> enforcement
> where we want to guarantee that a specific bandwidth is available on the 
> selected host
> for a vm to consume. From an ovs/vpp/linux bridge perspective this would 
> likely be tracked at
> the physnet level so when selecting a host we would want to ensure that the 
> physent
> is both available from the host and has enough bandwidth available to resever 
> for the instance.


OK, thanks, this is excellent information.

New question: will the placement service always be able to pick an acceptable 
provider, given that that the request needs X amount of bandwidth? IOW, are 
there other considerations besides quantitative amount (and possibly traits for 
qualitative concerns) that placement simply doesn’t know about? The example I 
have in mind is the case of stack vs. spread, where there are a few available 
providers that can meet the request. The logic for which one to pick can’t be 
in placement, though, as it’s a detail of the calling service. In the case of 
Nova, the assignment of VFs on vNICs usually should be spread, but that is not 
what placement knows, it’s handled by filters/weighers in Nova’s scheduler.

OK, that was a really long way of asking: will Neutron ever need to be able to 
determine the “best” choice from a selection of resource providers? Or will the 
fact that a resource provider has enough of a given resource be all that is 
needed?


-- Ed Leafe








-- Ed Leafe








-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Allocating Complex Resources

[Edward Leafe]

On Jun 7, 2017, at 1:44 PM, Mooney, Sean K > wrote:

> [Mooney, Sean K] neutron will need to use nested resource providers to track
> Network backend specific consumable resources in the future also. One example 
> is
> is hardware offloaded virtual(e.g. vitio/vhost-user) interfaces which due to
> their hardware based implementation are both a finite consumable
> resources and have numa affinity and there for need to track and nested.
> 
> Another example for neutron would be bandwidth based scheduling / sla 
> enforcement
> where we want to guarantee that a specific bandwidth is available on the 
> selected host
> for a vm to consume. From an ovs/vpp/linux bridge perspective this would 
> likely be tracked at
> the physnet level so when selecting a host we would want to ensure that the 
> physent
> is both available from the host and has enough bandwidth available to resever 
> for the instance.


OK, thanks, this is excellent information.

New question: will the placement service always be able to pick an acceptable 
provider, given that that the request needs X amount of bandwidth? IOW, are 
there other considerations besides quantitative amount (and possibly traits for 
qualitative concerns) that placement simply doesn’t know about? The example I 
have in mind is the case of stack vs. spread, where there are a few available 
providers that can meet the request. The logic for which one to pick can’t be 
in placement, though, as it’s a detail of the calling service. In the case of 
Nova, the assignment of VFs on vNICs usually should be spread, but that is not 
what placement knows, it’s handled by filters/weighers in Nova’s scheduler.

OK, that was a really long way of asking: will Neutron ever need to be able to 
determine the “best” choice from a selection of resource providers? Or will the 
fact that a resource provider has enough of a given resource be all that is 
needed?


-- Ed Leafe








-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Allocating Complex Resources

[Edward Leafe]

On Jun 6, 2017, at 9:56 AM, Chris Dent  wrote:
> 
> For clarity and completeness in the discussion some questions for
> which we have explicit answers would be useful. Some of these may
> appear ignorant or obtuse and are mostly things we've been over
> before. The goal is to draw out some clear statements in the present
> day to be sure we are all talking about the same thing (or get us
> there if not) modified for what we know now, compared to what we
> knew a week or month ago.


One other question that came up: do we have any examples of any service (such 
as Neutron or Cinder) that would require the modeling for nested providers? Or 
is this confined to Nova?


-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][placement] Allocating Complex Resources

[Edward Leafe]

On Jun 6, 2017, at 4:14 AM, Sylvain Bauza  wrote:
> 
> The Plan A option you mention hides the complexity of the
> shared/non-shared logic but to the price that it would make scheduling
> decisions on those criteries impossible unless you put
> filtering/weighting logic into Placement, which AFAIK we strongly
> disagree with.


Not necessarily. Well, not now, for sure, but that’s why we need Traits to be 
integrated into Flavors as soon as possible so that we can make requests with 
qualitative requirements, not just quantitative. When that work is done, we can 
add traits to differentiate local from shared storage, just like we have traits 
to distinguish HDD from SSD. So if a VM with only local disk is needed, that 
will be in the request, and placement will never return hosts with shared 
storage. 

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][scheduler] Anyone relying on the host_subset_size config option?

[Edward Leafe]

[resending to include the operators list]

The host_subset_size configuration option was added to the scheduler to help 
eliminate race conditions when two requests for a similar VM would be processed 
close together, since the scheduler’s algorithm would select the same host in 
both cases, leading to a race and a likely failure to build for the second 
request. By randomly choosing from the top N hosts, the likelihood of a race 
would be reduced, leading to fewer failed builds.

Current changes in the scheduling process now have the scheduler claiming the 
resources as soon as it selects a host. So in the case above with 2 similar 
requests close together, the first request will claim successfully, but the 
second will fail *while still in the scheduler*. Upon failing the claim, the 
scheduler will simply pick the next host in its weighed list until it finds one 
that it can claim the resources from. So the host_subset_size configuration 
option is no longer needed.

However, we have heard that some operators are relying on this option to help 
spread instances across their hosts, rather than using the RAM weigher. My 
question is: will removing this randomness from the scheduling process hurt any 
operators out there? Or can we safely remove that logic?


-- Ed Leafe


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][scheduler] Anyone relying on the host_subset_size config option?

[Edward Leafe]

The host_subset_size configuration option was added to the scheduler to help 
eliminate race conditions when two requests for a similar VM would be processed 
close together, since the scheduler’s algorithm would select the same host in 
both cases, leading to a race and a likely failure to build for the second 
request. By randomly choosing from the top N hosts, the likelihood of a race 
would be reduced, leading to fewer failed builds.

Current changes in the scheduling process now have the scheduler claiming the 
resources as soon as it selects a host. So in the case above with 2 similar 
requests close together, the first request will claim successfully, but the 
second will fail *while still in the scheduler*. Upon failing the claim, the 
scheduler will simply pick the next host in its weighed list until it finds one 
that it can claim the resources from. So the host_subset_size configuration 
option is no longer needed.

However, we have heard that some operators are relying on this option to help 
spread instances across their hosts, rather than using the RAM weigher. My 
question is: will removing this randomness from the scheduling process hurt any 
operators out there? Or can we safely remove that logic?


-- Ed Leafe


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] Active or passive role with our database layer

[Edward Leafe]

On May 23, 2017, at 1:43 PM, Jay Pipes  wrote:

> [1] Witness the join constructs in Golang in Kubernetes as they work around 
> etcd not being a relational data store:


Maybe it’s just me, but I found that Go code more understandable than some of 
the SQL we are using in the placement engine. :)

I assume that the SQL in a relational engine is faster than the same thing in 
code, but is that difference significant? For extremely large data sets I think 
that the database processing may be rate limiting, but is that the case here? 
Sometimes it seems that we are overly obsessed with optimizing data handling 
when the amount of data is relatively small. A few million records should be 
fast enough using just about anything.

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][scheduler] Discussion on how to do claims

[Edward Leafe]

I missed the summit, so I also missed out on some of the decisions that were 
made. I don’t feel that some of them were ideal, and in talking to the people 
involved I’ve gotten various degrees of certainty about how settled things 
were. So I’ve not only pushed a series of patches as POC code [0] for my 
approach, but I’ve written a summary of my concerns [1].

Speaking with a few people today, since some people are not around, and I’m 
leaving for PyCon tomorrow, we felt it would be worthwhile to have everyone 
read this, and review the current proposed code by Sylvain [2], my code, and 
the blog post summary. Next week when we are all back in town we can set up a 
time to discuss, either in IRC or perhaps a hangout.

[0] https://review.openstack.org/#/c/464086/
[1] https://blog.leafe.com/claims-in-the-scheduler/
[2] https://review.openstack.org/#/c/460177/8 


-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Edward Leafe]

On May 15, 2017, at 9:00 PM, Flavio Percoco  wrote:

> [huge snip]

Thank you! We don’t need 50K of repeated text in every response.

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Scheduler meeting canceled for next Monday

[Edward Leafe]

Due to most participants being at the Forum this coming week, we will not hold 
our weekly Scheduler sub team meeting on Monday, May 8. Please join us the 
following Monday (May 15) in #openstack-meeting-alt at 1400 UTC.


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [api-wg][all] API-WG Liaison Update request

[Edward Leafe]

An important step in the process of creating API guidelines is getting feedback 
from the projects that will be affected by those guidelines before we approve 
them. To that end, we have encouraged projects to appoint a liaison, and when a 
guideline is approved by the WG, these liaisons are added to the review, and 
thus receive an email from Gerrit, which ideally should result in the liaisons 
reviewing the proposed change to either approve or express whatever concern 
they may have. Only when we have buy-in from the liaisons do we go ahead and 
merge the guideline.

That has worked reasonably well, but lately there have been some concerns. One 
such concern is that many of the liaisons may no longer be in the same position 
they were when they signed up. If that is the situation for you, please let me 
know so we can update our list and start looking for someone else to represent 
your project. We also don’t have a way to directly contact liaisons except 
through Gerrit, and that is not ideal. So we’d like to update the liaison list 
[0] with the email address for each liaison, so that we can reach out to you 
when needed. I’ve included a list of the current liaisons   below my sig.

So if you are a liaison, please reply to me (off-list, to reduce spam) with 
your project, whether you are able to continue in the liaison role, and your 
email address. I will update the official list, and then start the process of 
finding replacements for those who have moved on.

[0] http://specs.openstack.org/openstack/api-wg/liaisons.html 


-- Ed Leafe

Barbican: Douglas Mendizábal
Ceilometer: Chris Dent
Cinder: Scott DAngelo
Congress: Masahito Muroi
Designate: 
Glance: Stuart McLaren
Glance: Nikhil Komawar
Heat: Ryan Brown
Horizon: Cindy Lu
Ironic: Vladyslav Drok
Keystone: David Stanek
MagnetoDB: Ilya Sviridov
Magnum: Eli Qiao
Magnum: Hua Wang
Manila: Alex Meade
Manila: Goutham Pacha Ravi
Mistral: Renat Akhmerov
Murano: Nikolay Starodubtsev
Neutron: Henry Gessau
Nova: Matthew Gilliard
Nova: Alex Xu
Rally: 
Sahara: Michael McCune
Senlin: Qiming Teng
Swift: John Dickinson
Trove: Peter Stachowski
Trove: Amrith Kumar
Tripleo: 
Zaqar: Fei Long Wang

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] [elections] Available time and top priority

[Edward Leafe]

On Apr 10, 2017, at 1:52 PM, Matt Riedemann  wrote:
> 
> How will the TC grow a diverse membership if it's not even held, at least 
> every other week, in a timezone where the other half of the world can attend?

+1

-- Ed__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api]

[Edward Leafe]

On Nov 14, 2016, at 3:14 PM, Ed Leafe  wrote:

> Since we already had ‘new’, I thought that ‘nin’ would be consistent. No 
> other reason to prefer it, though.

s/new/neq

Damn you autocorrect!

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [elections][tc]Thoughts on the TC election process

[Edward Leafe]

On Oct 3, 2016, at 1:39 PM, Clint Byrum  wrote:
> 
> Of course, I read the essays of those who I don't know more carefully, and
> I often go searching through my ML archives to see if we've interacted on
> threads in the past. Still, I'm very unlikely to rank somebody higher than
> a personal acquaintance unless I don't have many personal acquaintances
> that I agree with that are nominated.

I understand, and would be less than honest if I stated that I never do the 
same thing. I'm questioning, though, whether that's a good thing, or part of 
our tribal nature as humans.

> So I get where you're going, but I don't think that can be "the
> election". In addition to not allowing me to judge peoples' character,
> it also introduces a _massive_ fraud incentive. If you are a candidate,
> and you write a paper that is equal to all others, you can gain votes just
> by secretly telling your friends which one is yours, and their implicit
> bias will 1) make many think this is morally ok, because they know you're
> a good candidate, and 2) make them more likely to vote for you.

I suppose that's a possible downside, although I don't know that anyone who 
would do that would have enough people they could call "friends" to get them 
elected. I know that such a tactic would certainly backfire if someone tried to 
get me to vote for them.

> One way to counter the problems associated with the popularity contest
> is to have some appointed members. We can admonish the TC to appoint a
> nominee who did not win the most votes, but who is more likely to break
> a groupthink cycle. This would only work if people paid attention to TC
> voting records, which AFAIK, nobody really does.

Heh, let's change the bylaws so that the top 4 (or 5 in the next cycle) 
vote-getters win, and the other two seats are chosen by lottery from the bottom 
5 vote getters! That's sure to liven things up! 

(kidding, of course!)

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [elections][tc]Thoughts on the TC election process

[Edward Leafe]

On Oct 3, 2016, at 12:18 PM, Clay Gerrard  wrote:
> 
>> After the nominations close, the election officials will assign each 
>> candidate a non-identifying label, such as a random number, and those 
>> officials will be the only ones who know which candidate is associated with 
>> which number. 
>> 
> I'm really uneasy about this suggestion.  Especially when it comes to 
> re-election, for the purposes of accountability I think it's really important 
> that voters be able to identify the candidates.  For some people there's a 
> difference in what they say and what they end up doing when left calling 
> shots from the bubble for too long.

This was a concern of mine, too, but IMO there haven't been too many cases 
where a TC member has said they would support X and then fail to do so. They 
might not prevail, being one of 13, but when that issue came up they were 
almost always consistent with what they said.

> As far as the other stuff... idk if familiarity == bias.  I'm sure lots of 
> occasions people vote for people they know because they *trust* them; but I 
> don't think that's bias?  I think a more common problem is when people vote 
> for a *name* they recognize without really knowing that person or what 
> they're about.  Or perhaps just as bad - *not* voting because they realize 
> they have on context to consider these candidates beyond name familiarity and 
> an (optional) email.

I think that with so many candidates for so few seats, most people simply don't 
have the time or the interest to look very deeply into things. I know that that 
shows up in the voting. Take the election from a year ago: there were 619 votes 
cast for 19 candidates. Out of these:
- 35 ballots only voted for one candidate
- 102 ballots voted for three or fewer
- 175 didn't even bother to vote for 6
- only 159 bothered to rank all the candidates

So I think that there is evidence that unless you are already well-known, most 
people aren't going to take the time to dig deeper. Maybe anonymous campaigns 
aren't the answer, but they certainly would help in this regard.

> I think a campaign period, and especially some effort [1] to have candidates 
> verbalize their viewpoints on topics that matter to the constituency could go 
> a long way towards giving people some more context beyond "i think this name 
> looks familiar; I don't really recognize this name"

Agreed 100%! It was made worse this year because the nominations closed on a 
Saturday, and with the late rush of people declaring their candidacy, gave no 
time at all for any sort of campaign discussions before voting began. There 
really needs to be a decent period of time allowed for people to get answers to 
whatever questions they may have.


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [elections][tc]Thoughts on the TC election process

[Edward Leafe]

On Oct 3, 2016, at 12:51 PM, Doug Hellmann  wrote:

> When I vote, I consider the positions a candidate takes, the ideas
> they propose, and -- equally importantly -- their track record of
> actually getting things done.  Hiding the candidate's identity makes
> it impossible to evaluate that track record and have a sense of
> whether they're likely to make any real progress on their ideas.

That's a very good point, and one I wrestled with. I tried to balance that with 
the desire to see an influx of new ideas; I guess this balance is where we 
differ.

> In the past we experimented with a few formal questions being posed
> to all candidates. I appreciate the fact that Gordon took the
> initiative and started a less formal thread on his own this time.
> I hope that everyone feels able to do the same, whether they have
> questions for specific candidates or for the entire slate.

I agree, and wish that the discussion Gordon started could have happened 
*before* people started voting. 

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [elections][tc]Thoughts on the TC election process

[Edward Leafe]

So the period of self-nominations for the Technical Committee seats has ended, 
and the voting has begun. I've been a very close observer of this process for 
several cycles, and I have some ideas I'd like to share. Full disclosure: I am 
a current candidate for the TC, and have been a candidate several times in the 
past, all of which were unsuccessful.

When deciding to run, candidates write a long, thoughtful essay on their 
reasons for wanting to serve on the TC, and those essays are typically the last 
you hear from them until the election. It has been rare for anyone to ask 
follow-up questions, or to challenge the candidates to explain their positions 
more definitively. I have spoken with many people at the Summits, which always 
closely followed the TC election (warning: unscientific samples ahead!), and 
what their selection process mostly boils down to is: they pick the names they 
are most familiar with. Many people don't read those long candidacy posts, and 
nearly all couldn't remember a single point that any of the candidates had put 
forth.

We are fortunate in that all of the candidates are exceptionally 
well-qualified, and those elected have put in excellent service while on the 
TC. But one thing I'm afraid of is that we tend to get into a situation where 
groupthink [0] is very likely. There are many excellent candidates running in 
every election, but it is rare for someone who hasn't been a PTL of a large 
project, and thus very visible, has been selected. Is this really the best 
approach?

I wrote a blog post about implicit bias [1], and in that post used the example 
of blind auditions for musical orchestras radically changing the selection 
results. Before the introduction of blind auditions, men overwhelmingly 
comprised orchestras, but once the people judging the auditions had no clue as 
to whether the musician was male or female, women began to be selected much 
more in proportion to their numbers in the audition pools. So I'd like to 
propose something for the next election: have candidates self-nominate as in 
the past, but instead of writing a big candidacy letter, just state their 
interest in serving. After the nominations close, the election officials will 
assign each candidate a non-identifying label, such as a random number, and 
those officials will be the only ones who know which candidate is associated 
with which number. The nomination period can be much, much shorter, and then 
followed by a week of campaigning (the part that's really missing in the 
current pro
 cess). Candidates will post their thoughts and positions, and respond to 
questions from people, and this is how the voters will know who best represents 
what they want to see in their TC.

The current candidacy essay would now be posted in the campaign period, rather 
than at the time of nomination, and should exclude the sort of biographical 
information that is currently the most important piece for many people. Keeping 
anonymity will be difficult, and will preclude the use of email for posting 
positions and responses, since email identifies the sender. So perhaps 
candidates could forward their posts to the election officials, who will post 
them for the candidates, identifying them by number only. The voting form will 
only list the candidate numbers, so the end result will be people casting votes 
for the candidates whose platform most matches what they want to see in the TC, 
and who have best answered any questions raised by others.

My feeling is that the result would be very different than the current process. 
My question, then, is whether that would be a good thing? It would require more 
work from the candidates and especially the election officials, so we should 
make sure that the goal is worth it. Do we want everyone to have an equal 
chance to serve on the TC, or should those who have earned name recognition by 
their excellent work in other parts of OpenStack continue to have an advantage? 

[0] https://en.wikipedia.org/wiki/Groupthink
[1] http://blog.leafe.com/bias/

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] open question to the candidates

[Edward Leafe]

On Oct 3, 2016, at 10:30 AM, gordon chung  wrote:

> the TC has historically been a reactive council that lets others ask for 
> change and acts as the final approver. do you believe the TC should be a 
> proactive committee that initiates change and if yes, to what scope? 
> more generally, what are some specific issues you'd like the TC address 
> in the coming year?

OK, I'm going to start with the standard cop-out answer: "It depends"

What I mean is that one of the duties of the TC is to be reactive: to act as a 
referee when there are issues brought to it. This can't and shouldn't change.

But I *would* like to see some more pro-active work, primarily in the area of 
technical matters. They are the "Technical" committee, after all. So while a 
heavy-handed, top-down approach is never going to work, there are areas where 
the TC must push all OpenStack projects forward. One area that they are already 
doing this is pushing projects to fully support Python 3. This is an essential 
technical matter, as Python 2 will be unsupported by 2020, and it is the TC's 
job to ensure that all of OpenStack is ready.

One other area where I would like to see the TC actively promote is in 
modernizing the architecture of the projects to keep up with the changes in the 
underlying technologies. Having been involved in the initial design decisions 
in 2010, I can state with certainty that had those same discussion been held in 
2016, we would have made very different choices. That's the nature of the world 
we operate in, and while change for the sake of change is a waste of time, 
change to keep OpenStack from becoming a outdated dinosaur is essential.

Tying those two points together brings me to a third: the expansion of 
languages used in OpenStack. We are and always have been a Python project. 
There were newer languages with some support by a few developers in the 
beginning, but as both Nova and Swift were Python, OpenStack was Python. And as 
things change, new languages will always come around that have some benefits 
that developers like. But experience tells me that every time you introduce a 
new language into the mix, you fracture the community. Yes, I know about 
Javascript in Horizon, but that's a particular case, as web browsers do not 
natively support Python. As long as we keep current with Python and its 
evolution, there is no good reason to fracture the development community within 
OpenStack. In the specific case of Swift and the use of Go, I wrote about my 
feelings here: http://blog.leafe.com/on_swift/ 

And thanks for posting this question. There is not nearly enough discussion 
when it comes to TC elections.

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] governance proposal worth a visit: Write down OpenStack principles

[Edward Leafe]

On Sep 8, 2016, at 8:13 AM, Chris Dent  wrote:

> The writing is starting from a detailed proposal which, as txx said in
> his response to me above, presents itself as a document that is "meant
> to document *existing* principles that we operate under but never
> documented properly. It's not really about a new set of rules, or really
> changing anything".
> 
> That is, it thinks of iself as an existing truth to be ratified.

I think that you've made an excellent case for exactly the sort of process that 
is being followed. Let me explain.

As Monty explained, the document is the result of the TC realizing that there 
were many things that were part of their shared understanding, but which were 
not necessarily things that a newcomer to OpenStack would realize. The document 
they are creating is simply a way to capture that tribal knowledge so that it 
is discoverable by everyone.

While I'm not a member of the TC, I have been closely following it for a few 
years now, and attend nearly every meeting. So I had some additional context 
that you might not have, and thus had a very different reaction to the 
document. The more that these things are recorded and available to all, the 
better it will be for the community, which is something that I know you 
violently agree with. :) This is just the first step in the documentation 
process of where we are today, so that everyone has a common starting point for 
discussing where we would like to be in the future.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

[Edward Leafe]

On Aug 27, 2016, at 12:18 PM, HU, BIN  wrote:

>> From telco perspective, those are the areas that allow innovation, and 
>> provide telco customers with new types of services.
> 
> We need innovation, starting from not limiting ourselves from bringing new 
> idea and new use cases, and bringing those impossibility to reality.

There is innovation in OpenStack, and there is innovation in things built on 
top of OpenStack. We are simply trying to keep the two layers from getting 
confused.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Some thoughts on API microversions

[Edward Leafe]

On Aug 4, 2016, at 8:18 AM, Andrew Laski  wrote:

> This gets to the point I'm trying to make. We don't guarantee old
> behavior in all cases at which point users can no longer rely on
> microversions to signal non breaking changes. And where we do guarantee
> old behavior sometimes we do it artificially because the only signal we
> have is microversions and that's the contract we're trying to adhere to.

I've always understood microversions to be a way to prevent breaking an 
automated tool when we change either the input or output of our API. Its 
benefit was less clear for the case of adding a new API, since there is no 
chance of breaking something that would never call it. We also accept that a 
bug fix doesn't require a microversion bump, as users should *never* be 
expecting a 5xx response, so not only does fixing that not need a bump, but 
such fixes can be backported to affect all microversions.

The idea that by specifying a distinct microversion would somehow guarantee an 
immutable behavior, though, is simply not the case. We discussed this at length 
at the midcycle regarding the dropping of the nova-network code; once that's 
dropped, there won't be any way to get that behavior no matter what 
microversion you specify. It's gone. We signal this with deprecation notices, 
release notes, etc., and it's up to individuals to move away from using that 
behavior during this deprecation period. A new microversion will never help 
anyone who doesn't follow these signals.

In the case that triggered this thread [0], the change was completely on the 
server side of things; no change to either the request or response of the API. 
It simply allowed a failed resize to be recovered more easily. That's a 
behavior change, not an API change, and frankly, I can't imagine anyone who 
would ever *want* the old behavior of leaving an instance in an error state. To 
me, that's not very different than fixing a 5xx response, as it is correcting 
an error on the server side.

So if you haven't guessed by now, I agree with Andrew that microversions are 
not the best choice for signaling all changes. I'm not sure that adding either 
of the things he proposed would address the issue in [0], though.

[0] http://lists.openstack.org/pipermail/openstack-dev/2016-August/100606.html


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][scheduler] Scheduler subteam meeting Monday at 1400 UTC

[Edward Leafe]

Sorry for the late email, but I was swamped last week after getting back from 
the mid-cycle. The next meeting of the Nova Scheduler subteam will be Monday, 
August 1, at 1400 UTC [0], in #openstack-meeting-alt

I've updated the agenda [1] with the main items; if you have other issues to 
discuss, please add them.

[0] http://www.timeanddate.com/worldclock/fixedtime.html?iso=20160801T14
[1] https://wiki.openstack.org/wiki/Meetings/NovaScheduler


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] [RFC] ResourceProviderTags - Manage Capabilities with ResourceProvider

[Edward Leafe]

On Jul 13, 2016, at 9:38 PM, Cheng, Yingxin  wrote:

>>Thinking about that a bit, it would seem that a host aggregate could also 
>> be represented as a namespace:name tag. That makes sense, since the fact 
>> that a host belongs to a particular aggregate is a qualitative aspect of 
>> that host.
>> 
> 
> Thanks for the feedback!
> 
> We’ve thought about the relationship between capability tags and host 
> aggregates carefully. And we decide not to blend it with host aggregates, for 
> several reasons below:
> 1. We want to manage capabilities in only ONE place, either in host 
> aggregates, compute_node records or with resource_provider records.
> 2. Compute services may need to attach discovered capabilities to its host. 
> It is inconvenient if we store caps with host aggregates, because 
> nova-compute needs to create/search host aggregates first, it can’t directly 
> attach caps.
> 3. Other services may need to attach discovered capabilities to its 
> resources. So the best place is to its related resource pool, not aggregates, 
> nor compute_node records. Note the relationship between resource pools and 
> host aggregates are N:N.
> 4. It’s logically correct to store caps with resource_providers, because caps 
> are actually owned by nodes or resource pools.
> 5. Scheduling will be faster if resource-providers are directly attached with 
> caps.
> 
> However, for user-defined caps, it still seems easier to manage them with 
> aggregates. We may want to manage them in a way different from pre-defined 
> caps. Or we can indirectly manage them through aggregates, but they are 
> actually stored with compute-node resource-providers in placement db.

Oh, I think you misunderstood me. Capabilities definitely belong with resource 
providers, not host aggregates, because not all RPs are hosts.

I'm thinking that host aggregates themselves are equivalent to capabilities for 
hosts. Imagine we have 10 hosts, and put 3 of them in an aggregate. How is that 
different than if we give those three a tag with the 'host_agg' namespace, and 
with tag named for the agg?

I'm just thinking out loud here. There might be opportunities to simplify a lot 
of the code between capability tags and host aggregates in the future, since it 
looks like host aggs are a structural subset of RP capability tags.

-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][nova-docker] Retiring nova-docker project

[Edward Leafe]

On Jul 7, 2016, at 8:33 PM, Joshua Harlow  wrote:
> 
> That's sad, how can we fix the fact that users/deployments have gone off into 
> their own silos and may be running their own forks; what went wrong (besides 
> some of the obvious stuff that I think I know about, that others probably 
> don't) that resulted in this happening?
> 
> Seems like something we can learn from by reflecting on this and trying to 
> find a path forward (or maybe we just accept there isn't one, idk).

I think admitting that it was a conceptual mis-match from the start would be 
the first step. Containers are not VMs. Attempts to treat them as such are 
always going to be poor compromises.

Containers have an important place in OpenStack; it's just not in Nova.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Questions about instance actions' update and finish

[Edward Leafe]

On Jun 29, 2016, at 10:05 PM, Matt Riedemann  wrote:
> 
>>> 2. The updated_at field is also empty, should we sync the updated_at
>>> time to the created_at time when we create the action and also update
>>> it whenever the action status changed, e.g finished.
>> 
>> When a finish_time is recorded that should definitely also update
>> updated_at. I would be in favor of having updated_at set when the
>> instance action is created. I've never fully understood why Nova doesn't
>> do that generally.
> 
> As discussed in the API meeting this morning, I thought it would be odd to 
> set updated_at = created_at when the record is created.

It's really very common. Think of 'updated_at' as meaning 'the last time this 
record was modified'. For a new record, the initial creation is also the last 
time it was modified.

-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Version header for OpenStack microversion support

[Edward Leafe]

On Jun 18, 2016, at 9:03 AM, Clint Byrum  wrote:

> Whatever API version is used behind the compute API is none of the user's
> business.

Actually, yeah, it is.

If I write an app or a tool that expects to send information in a certain 
format, and receive responses in a certain format, I don't want that to change 
when the cloud operator upgrades their system. I only want things to change 
when I specifically request that they change by specifying a new microversion.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Edward Leafe]

On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:

> But, if we add another possible state on the defcore side like conditional 
> pass,
> warning, yellow, etc. (the name doesn't matter) which is used to indicate that
> things on product X could only pass when strict validation was disabled (and
> be clear about where and why) then my concerns would be alleviated. I just do
> not want this to end up not being visible to end users trying to evaluate
> interoperability of different clouds using the test results.

+1

Don't fail them, but don't cover up their incompatibility, either.

-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] supporting Go

[Edward Leafe]

On May 9, 2016, at 1:58 PM, Hayes, Graham  wrote:

> This is not a "Go seems cool - lets go try that" decision from us - we
> know we have a performance problem with one of our components, and we
> have come to the conclusion that Go (or something like it) is the
> solution.

Whenever I hear claims that Python is “too slow for X”, I wonder what’s so 
special about X that makes it so much more demanding than, say, serving up 
YouTube. YouTube is written nearly entirely in Python, and has been for many, 
many years. The only parts that aren’t are those that were identified as 
particular performance bottlenecks, such as some parts of the encoding process. 
These were then written in C, which is drop-in compatible with Python using 
ctypes.


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][scheduler] Next Scheduler sub-team meeting

[Edward Leafe]

The next meeting for the Scheduler sub-team will be on Monday, May 9 at 1400 
UTC (http://www.timeanddate.com/worldclock/fixedtime.html?iso=20160509T14)

The agenda for the meeting is here; please add any items that you wish to 
discuss: 
https://wiki.openstack.org/wiki/Meetings/NovaScheduler#Agenda_for_next_meeting


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][osc] Use of openstack client for admin commands

[Edward Leafe]

On May 4, 2016, at 3:53 PM, Jay Pipes  wrote:

> My position is that if it's an HTTP API (as opposed to something like a 
> sqlalchemy-migrate sync command) then it belongs in a client that speaks the 
> OpenStack HTTP APIs. That is OSC as far as I can tell. I don't see a 
> difference between "admin" commands and "standard" commands.

Exactly. If it's an admin command and you're not an admin, you get an error. No 
need for a separate client.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On May 3, 2016, at 7:05 PM, Mark Doffman  wrote:

> This thread has been a depressing read.
> 
> I understand that the content is supposed to be distributed databases but for 
> me it has become an inquisition of cellsV2.

Thanks for bringing this up, and I feel a lot of the responsibility for this 
direction. To make how I see things clearer, I wrote a follow-up blog post [0], 
but for those who aren’t inclined to read it, I think that Cells V2 is a great 
idea and could be very helpful for many deployments. My only concern was the 
choice of fragmenting the data. I would hope that any further discussion 
focuses on that.

[0] http://blog.leafe.com/index.php/2016/05/04/mea-culpa-and-clarification/


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On May 3, 2016, at 6:45 AM, Miles Gould  wrote:

>> This DB could be an RDBMS or Cassandra, depending on the deployer's 
>> preferences
> AFAICT this would mean introducing and maintaining a layer that abstracts 
> over RDBMSes and Cassandra. That's a big abstraction, over two quite 
> different systems, and it would be hard to write code that performs well in 
> both cases. If performance in this layer is critical, then pick whichever DB 
> architecture handles the expected query load better and use that.

Agreed - you simply can’t structure the data the same way. When I read 
criticisms of Cassandra that include “you can’t do joins” or “you can’t 
aggregate”, it highlights this fact: you have to think about (and store) your 
data completely differently than you would in an RDBMS. You cannot simply 
abstract out the differences.

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On May 2, 2016, at 10:51 AM, Mike Bayer  wrote:

>> Concretely, we think that there are three possible approaches:
>> 1) We can use the SQLAlchemy API as the common denominator between a 
>> relational and non-relational implementation of the db.api component. These 
>> two implementation could continue to converge by sharing a large amount of 
>> code.
>> 2) We create a new non-relational implementation (from scratch) of the 
>> db.api component. It would require probably more work.
>> 3) We are also studying a last alternative: writing a SQLAlchemy engine 
>> that targets NewSQL databases (scalability + ACID):
>>  - https://github.com/cockroachdb/cockroach
>>  - https://github.com/pingcap/tidb
> 
> Going with a NewSQL backend is by far the best approach here.   That way, 
> very little needs to be reinvented and the application's approach to data 
> doesn't need to dramatically change.

I’m glad that Matthieu responded, but I did want to emphasize one thing: of 
*course* this isn’t an ideal approach, but it *is* a practical one. The biggest 
problem in any change like this isn’t getting it to work, or to perform better, 
or anything else except being able to make the change while disrupting as 
little of the existing code as possible. Taking an approach that would be more 
efficient would be a non-starter since it wouldn’t provide a clean upgrade path 
for existing deployments.

By getting this working without ripping out all of the data models that 
currently exist is an amazing feat. And if by doing so it shows that a 
distributed database is indeed possible, it’s done more than anything else that 
has ever been discussed in the past few years. 


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On Apr 28, 2016, at 5:35 PM, Clint Byrum  wrote:

> - Vitess [2] is a proven technology that serves _every_ request to
>  Youtube, and provides a familiar SQL interface with sharding built
>  in. Shard by project ID and you can just use regular index semantics.
>  Or if that's unacceptable (IMO it's fine since Vitess provides enough
>  redundancy that one shard has plenty of failure-domain reliability),
>  you can also use the built-in Hadoop support they have for doing
>  exactly what has been described (merge sorting the result of cross-cell
>  queries).

Thanks for that reference. I hadn’t heard of Vitess before, but it looks pretty 
capable.

> So, I have to ask, why is cells v2 being pushed so hard without looking
> outside OpenStack for actual existing solutions, which, IMO, are
> _numerous_, battle hardened, and simpler than cells.

Cells are a great concept, but of course the devil is in the implementation. So 
if having cells is an advantage (and that is a separate discussion that already 
seems settled), then we should focus on the best way to implement it for 
(short-term) efficiency and (long-term) maintainability.

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On Apr 28, 2016, at 1:09 PM, Jay Pipes  wrote:

> nova list as an admin (or any user frankly) should be a proxy call to Project 
> Searchlight and elasticsearch.
> 
> elasticsearch is a great interface for this kind of operation. We should use 
> it.

Oh, that’s great! A Java-based dependency! I’m *sure* that there will be no 
objection to that.

> The Cells architecture, which allows the operator to both scale the message 
> queue *and* limit the scope of failure domains, is a good one. Having a 
> database that stores only the local (to the cell) information is perfectly 
> fine given the top-level API database's index/mapping tables. Where this 
> design has short-comings, as Ed and others point out, are things like doing 
> list operations across dozens of separate cells. So, let's not use the Nova 
> database for that and instead use a solution that works very well for it: 
> Project Searchlight.

What I’m hearing is: let’s shard the database along a random line that 
seriously impacts performance, and then add another layer to help mitigate the 
performance impact of that decision.

> And for those that have small clouds and/or don't want/need to install 
> elasticsearch, OK, cool, stick with a single cell and a single RDBMS instance.

Your own tests showed that a single RDBMS instance doesn’t even break a sweat 
under your test loads. I don’t see why we need to shard it in the first place, 
especially if in doing so we add another layer of complexity and another 
dependency in order to compensate for that choice. Cells are a useful concept, 
but this proposed implementation is adding way too much complexity and debt to 
make it worthwhile.


-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On Apr 24, 2016, at 3:28 PM, Robert Collins  wrote:

> For instance, the things I think are essential for a distributed
> database based datastore:
> - good single-machine developer story. Must not need a physical
> cluster to hack on OpenStack
> - deal gracefully with single node/rack/site failures (when deployed
> appropriately) - allow limiting failure domain impact
> - straightforward programming model: wrong uses should be obvious to reviewers
> - low latency performance with big datasets: e.g. nova list as an
> admin should be able to get the Nth page as rapidly as the 2nd or 3rd.
> - code to deliver that should be (approximately) no worse than the current 
> code

Agree on all of these points, as well as the rest of your post.

After several hallway track discussions, as well as yesterday’s Cells V2 
discussion, I’ve written a follow-up post: 

http://blog.leafe.com/index.php/2016/04/28/fragmented-data/

Feedback, of course, is welcomed!


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On Apr 24, 2016, at 3:28 PM, Robert Collins  wrote:

> 

Heh, glad you said it! :)

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On Apr 24, 2016, at 6:41 AM, Michael Bayer  wrote:

> I'm only seeking to counter what appears to be the premise of your blog post, 
> which is that distributed and SQL are mutually exclusive.   When you say, 
> "why don't we just query the database?"  You can make distributed SQL look 
> like that to the application as well, but it might not bring any advantages.  
>  

That’s my fault for the confusion. My problem with SQL was in the modeling of 
disparate resources, each of which has all sorts of different qualities, into a 
single relational model. I fully understand that SQL DBs can have a distributed 
model like that of key-value stores or column family stores.


-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Edward Leafe]

On Apr 23, 2016, at 3:10 PM, Jay Pipes > wrote:

> BTW, note to Ed Leafe... unless your distributed data store supports 
> transactional semantics, you can't use a distributed data store for these 
> types of solutions. Instead, you will need to write a whole bunch of code 
> that does post-auditing of claims and quotas and a system that accepts that 
> oversubscription and out-of-sync quota limits and usages is a fact of life. 

Yes, that was one of the things that I liked about Cassandra: 
http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0 


For instance, in the scheduler, making a claim would simply fail if another 
process had changed any of the resources in question, as the transaction would 
not be allowed.

> Not to mention needing to implement JOINs in Python.

Heh, JOIN in a column family database is not needed. You do have to think about 
data a lot differently, and that means unlearning everything you know about 
normalization. As a long-time SQL DB admin, it took my brain quite some time to 
be able to understand the different approach.

-- Ed Leafe__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Edward Leafe]

On Apr 23, 2016, at 10:10 AM, Thierry Carrez  wrote:

>> I think replacing nova's persistent storage layer with a distributed
>> database would have a great effect - but I do not think it would have
>> anything to do with the database itself. It would come from the act that
>> would be completely necessary to accomplish that- completely rewriting
>> the persistence layer.
> 
> So... The folks from the Discovery initiative working on a 
> massively-distributed cloud use case have been working on incremental changes 
> to make that use case better supported by stock OpenStack. That includes an 
> oslo.db driver backed by a distributed database. Being scientists, they ran 
> interesting experiments to see when and where it made sense.
> 
> They will present their findings in the upstream dev track, and I think it 
> makes a good data point for this discussion:
> 
> https://www.openstack.org/summit/austin-2016/summit-schedule/events/7342

That’s exactly the scenario I had in mind. When I first was tasked with 
creating cells in the early days of OpenStack, Rackspace wanted a global 
deployment of individual cells that could be addressed individually or as a 
single deployment. It wasn’t the load on the database that was ever the issue; 
it was the inability to keep the data synchronized across the globe. I pushed 
for something better than MySQL at this, but was not successful in convincing 
many others. I’m really looking forward to hearing the results of these tests.

-- Ed Leafe
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova-scheduler] Scheduler team meeting Monday, 2016.04.18 1400 UTC

[Edward Leafe]

Sorry for the late note, but busy weekend, etc.

The next Nova Scheduler meeting will be tomorrow (Monday) at 1400 UTC in 
#openstack-meeting-alt

http://www.timeanddate.com/worldclock/fixedtime.html?iso=20160418T14

The agenda is posted here: 
https://wiki.openstack.org/wiki/Meetings/NovaScheduler

Please note that there will not be a meeting next Monday, as most of us will be 
at the Austin Summit.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Newton midcycle planning

[Edward Leafe]

On Apr 12, 2016, at 6:07 PM, Bhandaru, Malini K  
wrote:

> Intel would be pleased to host the Nova midcycle meetup either at San 
> Antonio, Texas or Hillsboro, Oregon during R-15 (June 20-24) or R-11 (July 
> 18-22) as preferred by the Nova community.

In July? Oregon > San Antonio


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][scheduler] Next Meeting: Monday, April 4 @ 1400UTC

[Edward Leafe]

The next meeting for the Nova Scheduler sub-team is scheduled for Monday, April 
4 at 1400 UTC.

http://www.timeanddate.com/worldclock/fixedtime.html?iso=20160404T14

The agenda is at https://wiki.openstack.org/wiki/Meetings/NovaScheduler - feel 
free to add items that need to be discussed to that page.

I won't be able to make it, but Chris Dent has graciously volunteered to run 
the meeting. Be nice to him.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [election][TC] TC Candidacy

[Edward Leafe]

Greetings!

I am announcing my candidacy for the OpenStack Technical Committee. As a 
long-time developer, I have been part of projects that have succeeded and 
others that have not; in either event, I always learned something to apply to 
the next endeavor. I would like to use that experience to help guide OpenStack 
forward as part of the TC.

For those who may not know me, my name is Ed Leafe (edleafe on IRC), and I have 
been involved in OpenStack since the very beginning as part of the original 
Nova team. I am currently employed by IBM to work 100% of my time on upstream 
OpenStack, so I would have the freedom to devote as much time as needed to my 
TC duties. I have been participating in the TC meetings for over a year, and am 
familiar with the issues that come before it. I believe that I could contribute 
a lot more as a member of the TC, and that's why I'm asking for your vote.

Here are some of the issues that I would like to focus on:

* Adding clarity to the Big Tent

Opening up the OpenStack world to projects without having to go through the 
incubation and co-gating requirements was a huge step forward, but the feedback 
I've heard about the resulting mix is that it is confusing. A few months ago 
the TC added the 'starter-kit' tag to the baseline projects you would need to 
install to get OpenStack running; this was a good first step, but I think we 
need is a way to separate the "guts" of OpenStack from the parts that are built 
on top of that core. Is a project part of OpenStack itself? Or is it something 
that works on top of OpenStack (or any other cloud)? I'd like to see projects 
clearly separated into those that provide 'cloud' services, and those that work 
with those cloud pieces to make them easier to deploy/manage/report. Why is 
this distinction important? Because I feel that OpenStack should only have a 
single project for any particular service, and if someone has an idea for 
making it better, they need to work with the existing project, not compete. But 
in the world of projects built on top of OpenStack services, I say let's invite 
competition! There doesn't have to be a single "winner", as these will more 
likely tend to be solving particular use cases. Forcing these to be in a single 
project usually results in bloated, inefficient code.

* Improving the user experience

Part of my current job is mentoring fellow IBMers who are new to OpenStack in 
what it is, how to use it, etc. Have you ever tried to explain how to set up 
and configure OpenStack to anyone? Then you know just how difficult it is. 
That's one of the reasons I have been involved in groups such as the API 
working group, the Nova API team, and the Configuration Option cleanup efforts, 
as they represent places where OpenStack needs improvement. The recent efforts 
to open dialogs between ops and devs is a great start, and I'd certainly like 
to build on that. As a TC member, I would promote efforts to make all of 
OpenStack more manageable to deploy and maintain, since the coolest software in 
the world is useless if people can't get it running.

* Promoting consistency across OpenStack projects

There is some inconsistency within a single project like Nova, but when you 
work with multiple projects, the inconsistencies are glaring. And while the TC 
is not a strong-arm enforcer of standards, it does have considerable influence 
on how projects evolve, and I would like to see the TC push harder at 
establishing standards, as the API Working Group is doing, and then encouraging 
adoption of those standards across projects. The sanity of our operators is at 
stake!

In closing, I'd like to say that anyone who cares enough about OpenStack to 
want to be a part of the TC will certainly be worth your vote. I hope that I've 
presented enough to earn your vote.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [tc] "No Open Core" in 2016

[Edward Leafe]

On Feb 16, 2016, at 1:30 PM, Doug Hellmann  wrote:

> So I think the project team is doing everything we've asked.  We
> changed our policies around new projects to emphasize the social
> aspects of projects, and community interactions. Telling a bunch
> of folks that they "are not OpenStack" even though they follow those
> policies is rather distressing.  I think we should be looking for
> ways to say "yes" to new projects, rather than "no."

So if some group creates a 100% open project, and follows all of the Opens, at 
what point do we consider relevance to OpenStack itself? How about a 100% open 
text editor? Can we add that, since after all OpenStack code is written with 
text editors.

CDNs are not part of OpenStack, even if some parts of some projects may use one 
from time to time. A common interface to CDNs seems to address a problem that 
is not really part of what OpenStack does.

It's great to want to include people, but I think that there is more to it than 
just whether their practices mirror ours.

-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev