Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-20 Thread Joe Gordon
On Aug 19, 2014 10:45 AM, Day, Phil philip@hp.com wrote:

  -Original Message-
  From: Nikola Đipanov [mailto:ndipa...@redhat.com]
  Sent: 19 August 2014 17:50
  To: openstack-dev@lists.openstack.org
  Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible
Resource
  Tracking
 
  On 08/19/2014 06:39 PM, Sylvain Bauza wrote:
   On the other hand, ERT discussion is decoupled from the scheduler
   split discussion and will be delayed until Extensible Resource Tracker
   owner (Paul Murray) is back from vacation.
   In the mean time, we're considering new patches using ERT as
   non-acceptable, at least until a decision is made about ERT.
  
 
  Even though this was not officially agreed I think this is the least we
can do
  under the circumstances.
 
  A reminder that a revert proposal is up for review still, and I
consider it fair
  game to approve, although it would be great if we could hear from Paul
first:
 
https://review.openstack.org/115218

 Given the general consensus seemed to be to wait some before deciding
what to do here, isn't putting the revert patch up for approval a tad
premature ?

 The RT may be not able to cope with all of the new and more complex
resource types we're now trying to schedule, and so it's not surprising
that the ERT can't fix that.  It does however address some specific use
cases that the current RT can't cope with,  the spec had a pretty through
review under the new process, and was discussed during the last 2 design
summits.   It worries me that we're continually failing to make even small
and useful progress in this area.

 Sylvain's approach of leaving the ERT in place so it can be used for the
use cases it was designed for while holding back on doing some of the more
complex things than might need either further work in the ERT, or some more
fundamental work in the RT (which feels like as L or M timescales based on
current progress) seemed pretty pragmatic to me.

++, I really don't like the idea of rushing the revert of a feature that
went through significant design discussion especially when the author is
away and cannot defend it.


 Phil

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-20 Thread Nikola Đipanov
On 08/20/2014 08:27 AM, Joe Gordon wrote:
 
 On Aug 19, 2014 10:45 AM, Day, Phil philip@hp.com
 mailto:philip@hp.com wrote:

  -Original Message-
  From: Nikola Đipanov [mailto:ndipa...@redhat.com
 mailto:ndipa...@redhat.com]
  Sent: 19 August 2014 17:50
  To: openstack-dev@lists.openstack.org
 mailto:openstack-dev@lists.openstack.org
  Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible
 Resource
  Tracking
 
  On 08/19/2014 06:39 PM, Sylvain Bauza wrote:
   On the other hand, ERT discussion is decoupled from the scheduler
   split discussion and will be delayed until Extensible Resource Tracker
   owner (Paul Murray) is back from vacation.
   In the mean time, we're considering new patches using ERT as
   non-acceptable, at least until a decision is made about ERT.
  
 
  Even though this was not officially agreed I think this is the least
 we can do
  under the circumstances.
 
  A reminder that a revert proposal is up for review still, and I
 consider it fair
  game to approve, although it would be great if we could hear from
 Paul first:
 
https://review.openstack.org/115218

 Given the general consensus seemed to be to wait some before deciding
 what to do here, isn't putting the revert patch up for approval a tad
 premature ?

There was a recent discussion about reverting patches, and from that
(but not only) my understanding is that we should revert whenever in doubt.

Putting the patch back in is easy, and if proven wrong I'd be the first
to +2 it. As scary as they sound - I don't think reverts are a big deal.


 The RT may be not able to cope with all of the new and more complex
 resource types we're now trying to schedule, and so it's not surprising
 that the ERT can't fix that.  It does however address some specific use
 cases that the current RT can't cope with,  the spec had a pretty
 through review under the new process, and was discussed during the last
 2 design summits.   It worries me that we're continually failing to make
 even small and useful progress in this area.

 Sylvain's approach of leaving the ERT in place so it can be used for
 the use cases it was designed for while holding back on doing some of
 the more complex things than might need either further work in the ERT,
 or some more fundamental work in the RT (which feels like as L or M
 timescales based on current progress) seemed pretty pragmatic to me.
 
 ++, I really don't like the idea of rushing the revert of a feature that
 went through significant design discussion especially when the author is
 away and cannot defend it.
 

Fair enough - I will WIP the revert until Phil is back. It's the right
thing to do seeing that he is away.

However - I don't agree with using the length of discussion around the
feature as a valid argument against reverting.

I've supplied several technical arguments on the original thread to why
I think we should revert it, and would expect a discussion that either
refutes them, or provides alternative ways forward.

Saying 'but we talked about it at length' is the ultimate appeal to
imaginary authority and frankly not helping at all.

N.



 Phil

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 mailto:OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-20 Thread Jay Pipes

On 08/20/2014 04:48 AM, Nikola Đipanov wrote:

On 08/20/2014 08:27 AM, Joe Gordon wrote:

On Aug 19, 2014 10:45 AM, Day, Phil philip@hp.com
mailto:philip@hp.com wrote:



-Original Message-
From: Nikola Đipanov [mailto:ndipa...@redhat.com

mailto:ndipa...@redhat.com]

Sent: 19 August 2014 17:50
To: openstack-dev@lists.openstack.org

mailto:openstack-dev@lists.openstack.org

Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible

Resource

Tracking

On 08/19/2014 06:39 PM, Sylvain Bauza wrote:

On the other hand, ERT discussion is decoupled from the scheduler
split discussion and will be delayed until Extensible Resource Tracker
owner (Paul Murray) is back from vacation.
In the mean time, we're considering new patches using ERT as
non-acceptable, at least until a decision is made about ERT.



Even though this was not officially agreed I think this is the least

we can do

under the circumstances.

A reminder that a revert proposal is up for review still, and I

consider it fair

game to approve, although it would be great if we could hear from

Paul first:


   https://review.openstack.org/115218


Given the general consensus seemed to be to wait some before deciding

what to do here, isn't putting the revert patch up for approval a tad
premature ?


There was a recent discussion about reverting patches, and from that
(but not only) my understanding is that we should revert whenever in doubt.


Right.

http://lists.openstack.org/pipermail/openstack-dev/2014-August/042728.html


Putting the patch back in is easy, and if proven wrong I'd be the first
to +2 it. As scary as they sound - I don't think reverts are a big deal.


Neither do I. I think it's more appropriate to revert quickly and then 
add it back after any discussions, per the above revert policy.




The RT may be not able to cope with all of the new and more complex

resource types we're now trying to schedule, and so it's not surprising
that the ERT can't fix that.  It does however address some specific use
cases that the current RT can't cope with,  the spec had a pretty
through review under the new process, and was discussed during the last
2 design summits.   It worries me that we're continually failing to make
even small and useful progress in this area.


Sylvain's approach of leaving the ERT in place so it can be used for

the use cases it was designed for while holding back on doing some of
the more complex things than might need either further work in the ERT,
or some more fundamental work in the RT (which feels like as L or M
timescales based on current progress) seemed pretty pragmatic to me.

++, I really don't like the idea of rushing the revert of a feature that
went through significant design discussion especially when the author is
away and cannot defend it.


Fair enough - I will WIP the revert until Phil is back. It's the right
thing to do seeing that he is away.


Well, it's as much (or more?) Paul Murray and Andrea Rosa :)


However - I don't agree with using the length of discussion around the
feature as a valid argument against reverting.


Neither do I.


I've supplied several technical arguments on the original thread to why
I think we should revert it, and would expect a discussion that either
refutes them, or provides alternative ways forward.

Saying 'but we talked about it at length' is the ultimate appeal to
imaginary authority and frankly not helping at all.


Agreed. Perhaps it's just my provocative nature, but I hear a lot of 
we've already decided/discussed this talk especially around the 
scheduler and RT stuff, and I don't think the argument holds much water. 
We should all be willing to reconsider design decisions and discussions 
when appropriate, and in the case of the RT, this discussion is timely 
and appropriate due to the push to split the scheduler out of Nova 
(prematurely IMO).


Best,
-jay


N.




Phil

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

mailto:OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-20 Thread Daniel P. Berrange
On Wed, Aug 20, 2014 at 08:33:31AM -0400, Jay Pipes wrote:
 On 08/20/2014 04:48 AM, Nikola Đipanov wrote:
 On 08/20/2014 08:27 AM, Joe Gordon wrote:
 On Aug 19, 2014 10:45 AM, Day, Phil philip@hp.com
 mailto:philip@hp.com wrote:
 
 -Original Message-
 From: Nikola Đipanov [mailto:ndipa...@redhat.com
 mailto:ndipa...@redhat.com]
 Sent: 19 August 2014 17:50
 To: openstack-dev@lists.openstack.org
 mailto:openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible
 Resource
 Tracking
 
 On 08/19/2014 06:39 PM, Sylvain Bauza wrote:
 On the other hand, ERT discussion is decoupled from the scheduler
 split discussion and will be delayed until Extensible Resource Tracker
 owner (Paul Murray) is back from vacation.
 In the mean time, we're considering new patches using ERT as
 non-acceptable, at least until a decision is made about ERT.
 
 
 Even though this was not officially agreed I think this is the least
 we can do
 under the circumstances.
 
 A reminder that a revert proposal is up for review still, and I
 consider it fair
 game to approve, although it would be great if we could hear from
 Paul first:
 
https://review.openstack.org/115218
 
 Given the general consensus seemed to be to wait some before deciding
 what to do here, isn't putting the revert patch up for approval a tad
 premature ?
 
 There was a recent discussion about reverting patches, and from that
 (but not only) my understanding is that we should revert whenever in doubt.
 
 Right.
 
 http://lists.openstack.org/pipermail/openstack-dev/2014-August/042728.html
 
 Putting the patch back in is easy, and if proven wrong I'd be the first
 to +2 it. As scary as they sound - I don't think reverts are a big deal.
 
 Neither do I. I think it's more appropriate to revert quickly and then add
 it back after any discussions, per the above revert policy.
 
 
 The RT may be not able to cope with all of the new and more complex
 resource types we're now trying to schedule, and so it's not surprising
 that the ERT can't fix that.  It does however address some specific use
 cases that the current RT can't cope with,  the spec had a pretty
 through review under the new process, and was discussed during the last
 2 design summits.   It worries me that we're continually failing to make
 even small and useful progress in this area.
 
 Sylvain's approach of leaving the ERT in place so it can be used for
 the use cases it was designed for while holding back on doing some of
 the more complex things than might need either further work in the ERT,
 or some more fundamental work in the RT (which feels like as L or M
 timescales based on current progress) seemed pretty pragmatic to me.
 
 ++, I really don't like the idea of rushing the revert of a feature that
 went through significant design discussion especially when the author is
 away and cannot defend it.
 
 Fair enough - I will WIP the revert until Phil is back. It's the right
 thing to do seeing that he is away.
 
 Well, it's as much (or more?) Paul Murray and Andrea Rosa :)
 
 However - I don't agree with using the length of discussion around the
 feature as a valid argument against reverting.
 
 Neither do I.
 
 I've supplied several technical arguments on the original thread to why
 I think we should revert it, and would expect a discussion that either
 refutes them, or provides alternative ways forward.
 
 Saying 'but we talked about it at length' is the ultimate appeal to
 imaginary authority and frankly not helping at all.
 
 Agreed. Perhaps it's just my provocative nature, but I hear a lot of we've
 already decided/discussed this talk especially around the scheduler and RT
 stuff, and I don't think the argument holds much water. We should all be
 willing to reconsider design decisions and discussions when appropriate, and
 in the case of the RT, this discussion is timely and appropriate due to the
 push to split the scheduler out of Nova (prematurely IMO).

Yes, this is absolutely right. Even if we have approved a spec / blueprint
we *always* reserve the right to change our minds at a later date if new
information or points of view come to light. Hopefully this will be fairly
infrequent and we won't do it lightly, but it is a key thing we accept as
a possible outcome of the process we follow.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-20 Thread Nikola Đipanov
On 08/20/2014 02:33 PM, Jay Pipes wrote:
 On 08/20/2014 04:48 AM, Nikola Đipanov wrote:
 Fair enough - I will WIP the revert until Phil is back. It's the right
 thing to do seeing that he is away.
 
 Well, it's as much (or more?) Paul Murray and Andrea Rosa :)
 

Yes sorry - meant to say Paul but was replying to Phil :)

So - until Paul Murray is back.

N.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-20 Thread Sylvain Bauza


Le 20/08/2014 15:13, Daniel P. Berrange a écrit :

On Wed, Aug 20, 2014 at 08:33:31AM -0400, Jay Pipes wrote:

On 08/20/2014 04:48 AM, Nikola Đipanov wrote:

On 08/20/2014 08:27 AM, Joe Gordon wrote:

On Aug 19, 2014 10:45 AM, Day, Phil philip@hp.com
mailto:philip@hp.com wrote:

-Original Message-
From: Nikola Đipanov [mailto:ndipa...@redhat.com

mailto:ndipa...@redhat.com]

Sent: 19 August 2014 17:50
To: openstack-dev@lists.openstack.org

mailto:openstack-dev@lists.openstack.org

Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible

Resource

Tracking

On 08/19/2014 06:39 PM, Sylvain Bauza wrote:

On the other hand, ERT discussion is decoupled from the scheduler
split discussion and will be delayed until Extensible Resource Tracker
owner (Paul Murray) is back from vacation.
In the mean time, we're considering new patches using ERT as
non-acceptable, at least until a decision is made about ERT.


Even though this was not officially agreed I think this is the least

we can do

under the circumstances.

A reminder that a revert proposal is up for review still, and I

consider it fair

game to approve, although it would be great if we could hear from

Paul first:

   https://review.openstack.org/115218

Given the general consensus seemed to be to wait some before deciding

what to do here, isn't putting the revert patch up for approval a tad
premature ?

There was a recent discussion about reverting patches, and from that
(but not only) my understanding is that we should revert whenever in doubt.

Right.

http://lists.openstack.org/pipermail/openstack-dev/2014-August/042728.html


Putting the patch back in is easy, and if proven wrong I'd be the first
to +2 it. As scary as they sound - I don't think reverts are a big deal.

Neither do I. I think it's more appropriate to revert quickly and then add
it back after any discussions, per the above revert policy.


The RT may be not able to cope with all of the new and more complex

resource types we're now trying to schedule, and so it's not surprising
that the ERT can't fix that.  It does however address some specific use
cases that the current RT can't cope with,  the spec had a pretty
through review under the new process, and was discussed during the last
2 design summits.   It worries me that we're continually failing to make
even small and useful progress in this area.

Sylvain's approach of leaving the ERT in place so it can be used for

the use cases it was designed for while holding back on doing some of
the more complex things than might need either further work in the ERT,
or some more fundamental work in the RT (which feels like as L or M
timescales based on current progress) seemed pretty pragmatic to me.

++, I really don't like the idea of rushing the revert of a feature that
went through significant design discussion especially when the author is
away and cannot defend it.

Fair enough - I will WIP the revert until Phil is back. It's the right
thing to do seeing that he is away.

Well, it's as much (or more?) Paul Murray and Andrea Rosa :)


However - I don't agree with using the length of discussion around the
feature as a valid argument against reverting.

Neither do I.


I've supplied several technical arguments on the original thread to why
I think we should revert it, and would expect a discussion that either
refutes them, or provides alternative ways forward.

Saying 'but we talked about it at length' is the ultimate appeal to
imaginary authority and frankly not helping at all.

Agreed. Perhaps it's just my provocative nature, but I hear a lot of we've
already decided/discussed this talk especially around the scheduler and RT
stuff, and I don't think the argument holds much water. We should all be
willing to reconsider design decisions and discussions when appropriate, and
in the case of the RT, this discussion is timely and appropriate due to the
push to split the scheduler out of Nova (prematurely IMO).

Yes, this is absolutely right. Even if we have approved a spec / blueprint
we *always* reserve the right to change our minds at a later date if new
information or points of view come to light. Hopefully this will be fairly
infrequent and we won't do it lightly, but it is a key thing we accept as
a possible outcome of the process we follow.


While I totally understand the possibility to change our minds about an 
approved spec, I'm a bit worried about the timeline and how it can 
possibly impact deliveries.
In our case, the problem was raised in between 2 and 3 months after the 
design session occurred, and more importantly, 1 week before Feature 
Proposal Freeze.
As a consequence, it freezes the specification and we loose the 
possibility to split the Scheduler by Juno.


I know there are discussions about how Nova should handle feature 
delivery, I just want to make sure that this corner case is part of the 
talks. IMHO, we need to make sure at least all cores validate a spec 
(either explicitely

Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-20 Thread Day, Phil


 -Original Message-
 From: Daniel P. Berrange [mailto:berra...@redhat.com]
 Sent: 20 August 2014 14:13
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource
 Tracking
 
 On Wed, Aug 20, 2014 at 08:33:31AM -0400, Jay Pipes wrote:
  On 08/20/2014 04:48 AM, Nikola Đipanov wrote:
  On 08/20/2014 08:27 AM, Joe Gordon wrote:
  On Aug 19, 2014 10:45 AM, Day, Phil philip@hp.com
  mailto:philip@hp.com wrote:
  
  -Original Message-
  From: Nikola Đipanov [mailto:ndipa...@redhat.com
  mailto:ndipa...@redhat.com]
  Sent: 19 August 2014 17:50
  To: openstack-dev@lists.openstack.org
  mailto:openstack-dev@lists.openstack.org
  Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible
  Resource
  Tracking
  
  On 08/19/2014 06:39 PM, Sylvain Bauza wrote:
  On the other hand, ERT discussion is decoupled from the scheduler
  split discussion and will be delayed until Extensible Resource
  Tracker owner (Paul Murray) is back from vacation.
  In the mean time, we're considering new patches using ERT as
  non-acceptable, at least until a decision is made about ERT.
  
  
  Even though this was not officially agreed I think this is the
  least
  we can do
  under the circumstances.
  
  A reminder that a revert proposal is up for review still, and I
  consider it fair
  game to approve, although it would be great if we could hear from
  Paul first:
  
 https://review.openstack.org/115218
  
  Given the general consensus seemed to be to wait some before
  deciding
  what to do here, isn't putting the revert patch up for approval a
  tad premature ?
  
  There was a recent discussion about reverting patches, and from that
  (but not only) my understanding is that we should revert whenever in
 doubt.
 
  Right.
 
  http://lists.openstack.org/pipermail/openstack-dev/2014-August/042728.
  html
 
  Putting the patch back in is easy, and if proven wrong I'd be the
  first to +2 it. As scary as they sound - I don't think reverts are a big 
  deal.
 
  Neither do I. I think it's more appropriate to revert quickly and then
  add it back after any discussions, per the above revert policy.
 
  
  The RT may be not able to cope with all of the new and more complex
  resource types we're now trying to schedule, and so it's not
  surprising that the ERT can't fix that.  It does however address
  some specific use cases that the current RT can't cope with,  the
  spec had a pretty through review under the new process, and was
 discussed during the last
  2 design summits.   It worries me that we're continually failing to make
  even small and useful progress in this area.
  
  Sylvain's approach of leaving the ERT in place so it can be used
  for
  the use cases it was designed for while holding back on doing some
  of the more complex things than might need either further work in
  the ERT, or some more fundamental work in the RT (which feels like
  as L or M timescales based on current progress) seemed pretty
 pragmatic to me.
  
  ++, I really don't like the idea of rushing the revert of a feature
  ++that
  went through significant design discussion especially when the
  author is away and cannot defend it.
  
  Fair enough - I will WIP the revert until Phil is back. It's the
  right thing to do seeing that he is away.
 
  Well, it's as much (or more?) Paul Murray and Andrea Rosa :)
 
  However - I don't agree with using the length of discussion around
  the feature as a valid argument against reverting.
 
  Neither do I.
 
  I've supplied several technical arguments on the original thread to
  why I think we should revert it, and would expect a discussion that
  either refutes them, or provides alternative ways forward.
  
  Saying 'but we talked about it at length' is the ultimate appeal to
  imaginary authority and frankly not helping at all.
 
  Agreed. Perhaps it's just my provocative nature, but I hear a lot of
  we've already decided/discussed this talk especially around the
  scheduler and RT stuff, and I don't think the argument holds much
  water. We should all be willing to reconsider design decisions and
  discussions when appropriate, and in the case of the RT, this
  discussion is timely and appropriate due to the push to split the scheduler
 out of Nova (prematurely IMO).
 
 Yes, this is absolutely right. Even if we have approved a spec / blueprint we
 *always* reserve the right to change our minds at a later date if new
 information or points of view come to light. Hopefully this will be fairly
 infrequent and we won't do it lightly, but it is a key thing we accept as a
 possible outcome of the process we follow.
 
My point was more that reverting a patch that does meet the use cases it was 
designed to cover, even if there is something more fundamental that needs to be 
looked at to cover some new use cases that weren't considered at the time is 
the route to stagnation.   

It seems (unless

[openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-19 Thread Sylvain Bauza

Hi,

As it was also stated in 
http://www.stillhq.com/openstack/juno/12.html, we expect to finish 
the prerequisites for Scheduler split by Juno. That requires to merge 
two blueprints, one with the spec validated but patches still under 
review [1] and one with the spec still subject to debates [2]



During this cycle, we discussed a lot how to update Scheduler with 
Compute Node stats, and the proposal we made (with a +2 on the spec) was 
to make use of Extensible Resource Tracker [3] to make that work easily 
without having to create new DB or Objects fields.


That said, it seems Extensible Resource Tracker (ERT) is still subject 
to discussions [4], where a revert change has been proposed [5]


I can understand the concerns raised by that, but that's unfortunate 
that we discuss if ERT is good or not by 1 week before Feature Proposal 
Freeze, which will happen this Thursday. Indeed, while the changes have 
been proposed now some weeks ago, it requires to create new patches by 2 
days in order to remove ERT as a dependency from the patches. IIUC, 
removing ERT means, with the concern of scheduler split, to create a new 
DB compute_node field and a new ComputeNode Object attribute for each 
resource (aggregate, instance, flavor) related to the host : that will 
generate more work and while the interface will be quite good (one field 
for one type), it will have huge payload (one ComputeNode version more, 
migrations etc.)


In that condition, I can't hardly see how we can reach the target of 
merging all the bits by Juno. That's why I'm coming back to you, 
Nova-ists, to know your opinion and what kind of things you would like.



Yeah, I know that's life and life can be hard, but I'm just advocating 
advices for getting all of your attention.



-Sylvain


[1] https://review.openstack.org/82778 and 
https://review.openstack.org/104556
[2] 
https://review.openstack.org/#/q/status:open+topic:bp/isolate-scheduler-db,n,z

[3] https://review.openstack.org/#/c/109643/
[4] 
http://lists.openstack.org/pipermail/openstack-dev/2014-August/042709.html

[5] https://review.openstack.org/115218

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-19 Thread Sylvain Bauza


Le 19/08/2014 14:51, Sylvain Bauza a écrit :

Hi,

As it was also stated in 
http://www.stillhq.com/openstack/juno/12.html, we expect to finish 
the prerequisites for Scheduler split by Juno. That requires to merge 
two blueprints, one with the spec validated but patches still under 
review [1] and one with the spec still subject to debates [2]



During this cycle, we discussed a lot how to update Scheduler with 
Compute Node stats, and the proposal we made (with a +2 on the spec) 
was to make use of Extensible Resource Tracker [3] to make that work 
easily without having to create new DB or Objects fields.


That said, it seems Extensible Resource Tracker (ERT) is still subject 
to discussions [4], where a revert change has been proposed [5]


I can understand the concerns raised by that, but that's unfortunate 
that we discuss if ERT is good or not by 1 week before Feature 
Proposal Freeze, which will happen this Thursday. Indeed, while the 
changes have been proposed now some weeks ago, it requires to create 
new patches by 2 days in order to remove ERT as a dependency from the 
patches. IIUC, removing ERT means, with the concern of scheduler 
split, to create a new DB compute_node field and a new ComputeNode 
Object attribute for each resource (aggregate, instance, flavor) 
related to the host : that will generate more work and while the 
interface will be quite good (one field for one type), it will have 
huge payload (one ComputeNode version more, migrations etc.)


In that condition, I can't hardly see how we can reach the target of 
merging all the bits by Juno. That's why I'm coming back to you, 
Nova-ists, to know your opinion and what kind of things you would like.



Yeah, I know that's life and life can be hard, but I'm just advocating 
advices for getting all of your attention.





So, we held Gantt meeting today, and minutes are here 
http://eavesdrop.openstack.org/meetings/gantt/2014/gantt.2014-08-19-15.00.html
The concern I raised above has been heavily discussed there and we ended 
up deciding that we need to clean up the interfaces in between Scheduler 
and Resource Tracker (ie. cleary identify an API for modeling resources 
sent to Scheduler, not a bare JSON blob containing nested dicts)


So, long story short, we identified PoC from Jay Pipes [6] as a good 
starting point for the resource models and classes that would contain 
the resource usage. Jay and I volunteered for porting that model into 
Nova soon and see how it will integrate with existing.
That means that the validated effort on a scheduler library 
(bp/scheduler-lib, here [1]) are still considered for Juno (and need to 
be reviewed), while patches related to [2] (bp/isolate-scheduler-db) are 
considered on-hold until we identify the proper way, as said previously.


On the other hand, ERT discussion is decoupled from the scheduler split 
discussion and will be delayed until Extensible Resource Tracker owner 
(Paul Murray) is back from vacation.
In the mean time, we're considering new patches using ERT as 
non-acceptable, at least until a decision is made about ERT.


-Sylvain

[6] https://review.openstack.org/#/c/103598/


-Sylvain


[1] https://review.openstack.org/82778 and 
https://review.openstack.org/104556
[2] 
https://review.openstack.org/#/q/status:open+topic:bp/isolate-scheduler-db,n,z

[3] https://review.openstack.org/#/c/109643/
[4] 
http://lists.openstack.org/pipermail/openstack-dev/2014-August/042709.html

[5] https://review.openstack.org/115218

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-19 Thread Nikola Đipanov
On 08/19/2014 06:39 PM, Sylvain Bauza wrote:
 On the other hand, ERT discussion is decoupled from the scheduler split
 discussion and will be delayed until Extensible Resource Tracker owner
 (Paul Murray) is back from vacation.
 In the mean time, we're considering new patches using ERT as
 non-acceptable, at least until a decision is made about ERT.
 

Even though this was not officially agreed I think this is the least we
can do under the circumstances.

A reminder that a revert proposal is up for review still, and I consider
it fair game to approve, although it would be great if we could hear
from Paul first:

  https://review.openstack.org/115218

Thanks,

N.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource Tracking

2014-08-19 Thread Day, Phil
 -Original Message-
 From: Nikola Đipanov [mailto:ndipa...@redhat.com]
 Sent: 19 August 2014 17:50
 To: openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Nova] Scheduler split wrt Extensible Resource
 Tracking
 
 On 08/19/2014 06:39 PM, Sylvain Bauza wrote:
  On the other hand, ERT discussion is decoupled from the scheduler
  split discussion and will be delayed until Extensible Resource Tracker
  owner (Paul Murray) is back from vacation.
  In the mean time, we're considering new patches using ERT as
  non-acceptable, at least until a decision is made about ERT.
 
 
 Even though this was not officially agreed I think this is the least we can do
 under the circumstances.
 
 A reminder that a revert proposal is up for review still, and I consider it 
 fair
 game to approve, although it would be great if we could hear from Paul first:
 
   https://review.openstack.org/115218

Given the general consensus seemed to be to wait some before deciding what to 
do here, isn't putting the revert patch up for approval a tad premature ?
 
The RT may be not able to cope with all of the new and more complex resource 
types we're now trying to schedule, and so it's not surprising that the ERT 
can't fix that.  It does however address some specific use cases that the 
current RT can't cope with,  the spec had a pretty through review under the new 
process, and was discussed during the last 2 design summits.   It worries me 
that we're continually failing to make even small and useful progress in this 
area.

Sylvain's approach of leaving the ERT in place so it can be used for the use 
cases it was designed for while holding back on doing some of the more complex 
things than might need either further work in the ERT, or some more fundamental 
work in the RT (which feels like as L or M timescales based on current 
progress) seemed pretty pragmatic to me.

Phil

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev