Re: [openstack-dev] [nova] Stein PTG summary

[Jay Pipes]

On 09/27/2018 11:15 AM, Eric Fried wrote:

On 09/27/2018 07:37 AM, Matt Riedemann wrote:

On 9/27/2018 5:23 AM, Sylvain Bauza wrote:



On Thu, Sep 27, 2018 at 2:46 AM Matt Riedemann mailto:mriede...@gmail.com>> wrote:

     On 9/26/2018 5:30 PM, Sylvain Bauza wrote:
  > So, during this day, we also discussed about NUMA affinity and we
     said
  > that we could possibly use nested resource providers for NUMA
     cells in
  > Stein, but given we don't have yet a specific Placement API
     query, NUMA
  > affinity should still be using the NUMATopologyFilter.
  > That said, when looking about how to use this filter for vGPUs,
     it looks
  > to me that I'd need to provide a new version for the NUMACell
     object and
  > modify the virt.hardware module. Are we also accepting this
     (given it's
  > a temporary question), or should we need to wait for the
     Placement API
  > support ?
  >
  > Folks, what are you thoughts ?

     I'm pretty sure we've said several times already that modeling
NUMA in
     Placement is not something for which we're holding up the extraction.


It's not an extraction question. Just about knowing whether the Nova
folks would accept us to modify some o.vo object and module just for a
temporary time until Placement API has some new query parameter.
Whether Placement is extracted or not isn't really the problem, it's
more about the time it will take for this query parameter ("numbered
request groups to be in the same subtree") to be implemented in the
Placement API.
The real problem we have with vGPUs is that if we don't have NUMA
affinity, the performance would be around 10% less for vGPUs (if the
pGPU isn't on the same NUMA cell than the pCPU). Not sure large
operators would accept that :(

-Sylvain


I don't know how close we are to having whatever we need for modeling
NUMA in the placement API, but I'll go out on a limb and assume we're
not close.


True story. We've been talking about ways to do this since (at least)
the Queens PTG, but haven't even landed on a decent design, let alone
talked about getting it specced, prioritized, and implemented. Since
full NRP support was going to be a prerequisite in any case, and our
Stein plate is full, Train is the earliest we could reasonably expect to
get the placement support going, let alone the nova side. So yeah...


Given that, if we have to do something within nova for NUMA
affinity for vGPUs for the NUMATopologyFilter, then I'd be OK with that
since it's short term like you said (although our "short term"
workarounds tend to last for many releases). Anyone that cares about
NUMA today already has to enable the scheduler filter anyway.



+1 to this ^


Or, I don't know, maybe don't do anything and deal with the (maybe) 10% 
performance impact from the cross-NUMA main memory <-> CPU hit for 
post-processing of already parallel-processed GPU data.


In other words, like I've mentioned in numerous specs and in person, I 
really don't think this is a major problem and is mostly something we're 
making a big deal about for no real reason.


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Stein PTG summary

[Eric Fried]

On 09/27/2018 07:37 AM, Matt Riedemann wrote:
> On 9/27/2018 5:23 AM, Sylvain Bauza wrote:
>>
>>
>> On Thu, Sep 27, 2018 at 2:46 AM Matt Riedemann > > wrote:
>>
>>     On 9/26/2018 5:30 PM, Sylvain Bauza wrote:
>>  > So, during this day, we also discussed about NUMA affinity and we
>>     said
>>  > that we could possibly use nested resource providers for NUMA
>>     cells in
>>  > Stein, but given we don't have yet a specific Placement API
>>     query, NUMA
>>  > affinity should still be using the NUMATopologyFilter.
>>  > That said, when looking about how to use this filter for vGPUs,
>>     it looks
>>  > to me that I'd need to provide a new version for the NUMACell
>>     object and
>>  > modify the virt.hardware module. Are we also accepting this
>>     (given it's
>>  > a temporary question), or should we need to wait for the
>>     Placement API
>>  > support ?
>>  >
>>  > Folks, what are you thoughts ?
>>
>>     I'm pretty sure we've said several times already that modeling
>> NUMA in
>>     Placement is not something for which we're holding up the extraction.
>>
>>
>> It's not an extraction question. Just about knowing whether the Nova
>> folks would accept us to modify some o.vo object and module just for a
>> temporary time until Placement API has some new query parameter.
>> Whether Placement is extracted or not isn't really the problem, it's
>> more about the time it will take for this query parameter ("numbered
>> request groups to be in the same subtree") to be implemented in the
>> Placement API.
>> The real problem we have with vGPUs is that if we don't have NUMA
>> affinity, the performance would be around 10% less for vGPUs (if the
>> pGPU isn't on the same NUMA cell than the pCPU). Not sure large
>> operators would accept that :(
>>
>> -Sylvain
> 
> I don't know how close we are to having whatever we need for modeling
> NUMA in the placement API, but I'll go out on a limb and assume we're
> not close.

True story. We've been talking about ways to do this since (at least)
the Queens PTG, but haven't even landed on a decent design, let alone
talked about getting it specced, prioritized, and implemented. Since
full NRP support was going to be a prerequisite in any case, and our
Stein plate is full, Train is the earliest we could reasonably expect to
get the placement support going, let alone the nova side. So yeah...

> Given that, if we have to do something within nova for NUMA
> affinity for vGPUs for the NUMATopologyFilter, then I'd be OK with that
> since it's short term like you said (although our "short term"
> workarounds tend to last for many releases). Anyone that cares about
> NUMA today already has to enable the scheduler filter anyway.
> 

+1 to this ^

-efried

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Stein PTG summary

[Matt Riedemann]

On 9/27/2018 5:23 AM, Sylvain Bauza wrote:



On Thu, Sep 27, 2018 at 2:46 AM Matt Riedemann > wrote:


On 9/26/2018 5:30 PM, Sylvain Bauza wrote:
 > So, during this day, we also discussed about NUMA affinity and we
said
 > that we could possibly use nested resource providers for NUMA
cells in
 > Stein, but given we don't have yet a specific Placement API
query, NUMA
 > affinity should still be using the NUMATopologyFilter.
 > That said, when looking about how to use this filter for vGPUs,
it looks
 > to me that I'd need to provide a new version for the NUMACell
object and
 > modify the virt.hardware module. Are we also accepting this
(given it's
 > a temporary question), or should we need to wait for the
Placement API
 > support ?
 >
 > Folks, what are you thoughts ?

I'm pretty sure we've said several times already that modeling NUMA in
Placement is not something for which we're holding up the extraction.


It's not an extraction question. Just about knowing whether the Nova 
folks would accept us to modify some o.vo object and module just for a 
temporary time until Placement API has some new query parameter.
Whether Placement is extracted or not isn't really the problem, it's 
more about the time it will take for this query parameter ("numbered 
request groups to be in the same subtree") to be implemented in the 
Placement API.
The real problem we have with vGPUs is that if we don't have NUMA 
affinity, the performance would be around 10% less for vGPUs (if the 
pGPU isn't on the same NUMA cell than the pCPU). Not sure large 
operators would accept that :(


-Sylvain


I don't know how close we are to having whatever we need for modeling 
NUMA in the placement API, but I'll go out on a limb and assume we're 
not close. Given that, if we have to do something within nova for NUMA 
affinity for vGPUs for the NUMATopologyFilter, then I'd be OK with that 
since it's short term like you said (although our "short term" 
workarounds tend to last for many releases). Anyone that cares about 
NUMA today already has to enable the scheduler filter anyway.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Stein PTG summary

[Sylvain Bauza]

On Thu, Sep 27, 2018 at 2:46 AM Matt Riedemann  wrote:

> On 9/26/2018 5:30 PM, Sylvain Bauza wrote:
> > So, during this day, we also discussed about NUMA affinity and we said
> > that we could possibly use nested resource providers for NUMA cells in
> > Stein, but given we don't have yet a specific Placement API query, NUMA
> > affinity should still be using the NUMATopologyFilter.
> > That said, when looking about how to use this filter for vGPUs, it looks
> > to me that I'd need to provide a new version for the NUMACell object and
> > modify the virt.hardware module. Are we also accepting this (given it's
> > a temporary question), or should we need to wait for the Placement API
> > support ?
> >
> > Folks, what are you thoughts ?
>
> I'm pretty sure we've said several times already that modeling NUMA in
> Placement is not something for which we're holding up the extraction.
>
>
It's not an extraction question. Just about knowing whether the Nova folks
would accept us to modify some o.vo object and module just for a temporary
time until Placement API has some new query parameter.
Whether Placement is extracted or not isn't really the problem, it's more
about the time it will take for this query parameter ("numbered request
groups to be in the same subtree") to be implemented in the Placement API.
The real problem we have with vGPUs is that if we don't have NUMA affinity,
the performance would be around 10% less for vGPUs (if the pGPU isn't on
the same NUMA cell than the pCPU). Not sure large operators would accept
that :(

-Sylvain

-- 
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Stein PTG summary

[Matt Riedemann]

On 9/26/2018 5:30 PM, Sylvain Bauza wrote:
So, during this day, we also discussed about NUMA affinity and we said 
that we could possibly use nested resource providers for NUMA cells in 
Stein, but given we don't have yet a specific Placement API query, NUMA 
affinity should still be using the NUMATopologyFilter.
That said, when looking about how to use this filter for vGPUs, it looks 
to me that I'd need to provide a new version for the NUMACell object and 
modify the virt.hardware module. Are we also accepting this (given it's 
a temporary question), or should we need to wait for the Placement API 
support ?


Folks, what are you thoughts ?


I'm pretty sure we've said several times already that modeling NUMA in 
Placement is not something for which we're holding up the extraction.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Stein PTG summary

[Sylvain Bauza]

Thanks for the recap email, Mel. Just a question inline for all the people
that were in the room by Wednesday.

Le jeu. 27 sept. 2018 à 00:10, melanie witt  a écrit :

> Hello everybody,
>
> I've written up a high level summary of the discussions we had at the
> PTG -- please feel free to reply to this thread to fill in anything I've
> missed.
>
> We used our PTG etherpad:
>
> https://etherpad.openstack.org/p/nova-ptg-stein
>
> as an agenda and each topic we discussed was filled in with agreements,
> todos, and action items during the discussion. Please check out the
> etherpad to find notes relevant to your topics of interest, and reach
> out to us on IRC in #openstack-nova, on this mailing list with the
> [nova] tag, or by email to me if you have any questions.
>
> Now, onto the high level summary:
>
> Rocky retrospective
> ===
> We began Wednesday morning with a retro on the Rocky cycle and captured
> notes on this etherpad:
>
> https://etherpad.openstack.org/p/nova-rocky-retrospective
>
> The runways review process was seen as overall positive and helped get
> some blueprint implementations merged that had languished in previous
> cycles. We agreed to continue with the runways process as-is in Stein
> and use it for approved blueprints. We did note that we could do better
> at queuing important approved work into runways, such as
> placement-related efforts that were not added to runways last cycle.
>
> We discussed whether or not to move the spec freeze deadline back to
> milestone 1 (we used milestone 2 in Rocky). I have an action item to dig
> into whether or not the late breaking regressions we found at RC time:
>
> https://etherpad.openstack.org/p/nova-rocky-release-candidate-todo
>
> were related to the later spec freeze at milestone 2. The question we
> want to answer is: did a later spec freeze lead to implementations
> landing later and resulting in the late detection of regressions at
> release candidate time?
>
> Finally, we discussed a lot of things around project management,
> end-to-end themes for a cycle, and people generally not feeling they had
> clarity throughout the cycle about which efforts and blueprints were
> most important, aside from runways. We got a lot of work done in Rocky,
> but not as much of it materialized into user-facing features and
> improvements as it did in Queens. Last cycle, we had thought runways
> would capture what is a priority at any given time, but looking back, we
> determined it would be helpful if we still had over-arching
> goals/efforts/features written down for people to refer to throughout
> the cycle. We dove deeper into that discussion on Friday during the hour
> before lunch, where we came up with user-facing themes we aim to
> accomplish in the Stein cycle:
>
> https://etherpad.openstack.org/p/nova-ptg-stein-priorities
>
> Note that these are _not_ meant to preempt anything in runways, these
> are just 1) for my use as a project manager and 2) for everyone's use to
> keep a bigger picture of our goals for the cycle in their heads, to aid
> in their work and review outside of runways.
>
> Themes
> ==
> With that, I'll briefly mention the themes we came up with for the cycle:
>
> * Compute nodes capable to upgrade and exist with nested resource
> providers for multiple GPU types
>
> * Multi-cell operational enhancements: resilience to "down" or
> poor-performing cells and cross-cell instance migration
>
> * Volume-backed user experience and API hardening: ability to specify
> volume type during boot-from-volume, detach/attach of root volume, and
> volume-backed rebuild
>
> These are the user-visible features and functionality we aim to deliver
> and we'll keep tabs on these efforts throughout the cycle to keep them
> making progress.
>
> Placement
> =
> As usual, we had a lot of discussions on placement-related topics, so
> I'll try to highlight the main things that stand out to me. Please see
> the "Placement" section of our PTG etherpad for all the details and
> additional topics we discussed.
>
> We discussed the regression in behavior that happened when we removed
> the Aggregate[Core|Ram|Disk]Filters from the scheduler filters -- these
> filters allowed operators to set overcommit allocation ratios per
> aggregate instead of per host. We agreed on the importance of restoring
> this functionality and hashed out a concrete plan, with two specs needed
> to move forward:
>
> https://review.openstack.org/552105
> https://review.openstack.org/544683
>
> The other standout discussions were around the placement extraction and
> closing the gaps in nested resource providers. For the placement
> extraction, we are focusing on full support of an upgrade from
> integrated placement => extracted placement, including assisting with
> making sure deployment tools like OpenStack-Ansible and TripleO are able
> to support the upgrade. For closing the gaps in nested resource
> providers, there are many parts to it that are 

[openstack-dev] [nova] Stein PTG summary

[melanie witt]

Hello everybody,

I've written up a high level summary of the discussions we had at the 
PTG -- please feel free to reply to this thread to fill in anything I've 
missed.


We used our PTG etherpad:

https://etherpad.openstack.org/p/nova-ptg-stein

as an agenda and each topic we discussed was filled in with agreements, 
todos, and action items during the discussion. Please check out the 
etherpad to find notes relevant to your topics of interest, and reach 
out to us on IRC in #openstack-nova, on this mailing list with the 
[nova] tag, or by email to me if you have any questions.


Now, onto the high level summary:

Rocky retrospective
===
We began Wednesday morning with a retro on the Rocky cycle and captured 
notes on this etherpad:


https://etherpad.openstack.org/p/nova-rocky-retrospective

The runways review process was seen as overall positive and helped get 
some blueprint implementations merged that had languished in previous 
cycles. We agreed to continue with the runways process as-is in Stein 
and use it for approved blueprints. We did note that we could do better 
at queuing important approved work into runways, such as 
placement-related efforts that were not added to runways last cycle.


We discussed whether or not to move the spec freeze deadline back to 
milestone 1 (we used milestone 2 in Rocky). I have an action item to dig 
into whether or not the late breaking regressions we found at RC time:


https://etherpad.openstack.org/p/nova-rocky-release-candidate-todo

were related to the later spec freeze at milestone 2. The question we 
want to answer is: did a later spec freeze lead to implementations 
landing later and resulting in the late detection of regressions at 
release candidate time?


Finally, we discussed a lot of things around project management, 
end-to-end themes for a cycle, and people generally not feeling they had 
clarity throughout the cycle about which efforts and blueprints were 
most important, aside from runways. We got a lot of work done in Rocky, 
but not as much of it materialized into user-facing features and 
improvements as it did in Queens. Last cycle, we had thought runways 
would capture what is a priority at any given time, but looking back, we 
determined it would be helpful if we still had over-arching 
goals/efforts/features written down for people to refer to throughout 
the cycle. We dove deeper into that discussion on Friday during the hour 
before lunch, where we came up with user-facing themes we aim to 
accomplish in the Stein cycle:


https://etherpad.openstack.org/p/nova-ptg-stein-priorities

Note that these are _not_ meant to preempt anything in runways, these 
are just 1) for my use as a project manager and 2) for everyone's use to 
keep a bigger picture of our goals for the cycle in their heads, to aid 
in their work and review outside of runways.


Themes
==
With that, I'll briefly mention the themes we came up with for the cycle:

* Compute nodes capable to upgrade and exist with nested resource 
providers for multiple GPU types


* Multi-cell operational enhancements: resilience to "down" or 
poor-performing cells and cross-cell instance migration


* Volume-backed user experience and API hardening: ability to specify 
volume type during boot-from-volume, detach/attach of root volume, and 
volume-backed rebuild


These are the user-visible features and functionality we aim to deliver 
and we'll keep tabs on these efforts throughout the cycle to keep them 
making progress.


Placement
=
As usual, we had a lot of discussions on placement-related topics, so 
I'll try to highlight the main things that stand out to me. Please see 
the "Placement" section of our PTG etherpad for all the details and 
additional topics we discussed.


We discussed the regression in behavior that happened when we removed 
the Aggregate[Core|Ram|Disk]Filters from the scheduler filters -- these 
filters allowed operators to set overcommit allocation ratios per 
aggregate instead of per host. We agreed on the importance of restoring 
this functionality and hashed out a concrete plan, with two specs needed 
to move forward:


https://review.openstack.org/552105
https://review.openstack.org/544683

The other standout discussions were around the placement extraction and 
closing the gaps in nested resource providers. For the placement 
extraction, we are focusing on full support of an upgrade from 
integrated placement => extracted placement, including assisting with 
making sure deployment tools like OpenStack-Ansible and TripleO are able 
to support the upgrade. For closing the gaps in nested resource 
providers, there are many parts to it that are documented on the 
aforementioned PTG etherpads. By closing the gaps with nested resource 
providers, we'll open the door for being able to support minimum 
bandwidth scheduling as well.


Cells
=
On cells, the main discussions were around resiliency "down" and 
poor-performing cells and