Re: [openstack-dev] [nova] RequestSpec object and Instance model
Le 03/06/2015 15:15, Nikola Đipanov a écrit : On 06/02/2015 03:14 PM, Sylvain Bauza wrote: Hi, Currently working on implementing the RequestSpec object BP [1], I had some cool comments on my change here : https://review.openstack.org/#/c/145528/12/nova/objects/request_spec.py,cm Since we didn't discussed on how to persist that RequestSpec object, I think the comment is valuable. For the moment, the only agreed spec for persisting the object that we have is [2] but there is also a corollar here that means that we would have to persist more than the current fields https://review.openstack.org/#/c/169901/3/specs/liberty/approved/add-buildrequest-obj.rst,cm So, there are 2 possibilities : #1, we only persist the RequestSpec for the sole Scheduler and in that case, we can leave as it is - only a few fields from Instance are stored #2, we consider that RequestSpec can be used for more than just the Scheduler, and then we need to make sure that we will have all the instance fields then. So these are 2 possibilities if we agree that we need to make progress on the spec as is defined and merged now. What I was complaining yesterday is that we don't seem to have done enough of high level investigation into this stuff before embarking on writing a set of specs that then due to their format obscure the problems we are actually trying to solve. Since Nova is big, it's pretty hard to take the big picture of all the problems that we have and provide a spec which can be fine grained enough for making sure it will take in account the overall problem. At least, I'm seriously considering the RequestSpec object and how it is persisted as a first attempt to version a Scheduler API and provide an upgrade path for changing it. Work around the scheduler touches on a lot of issues that have only recently been noticed. While I am all for the incremental approach, it seems silly to completely disregard the issues we already know about. We should have a high level overview of the problems we know we want to solve, and then come up with an incremental way of solving them, but not without keeping an eye on the big picture at all times. True story. That's why the biggest question I have is "do I actually add more technical debt to the project by writing RequestSpec, or is it something that helps reducing the debt ?" So, that's why I'm very open to any comments explaining which kind of problem the current proposal could raise, and how we could prevent that. IIUC, your concern is about duplicating some information within the RequestSpec object which would be persisted. My point is to consider that if the usage is well defined (ie. only a contract between conductor and scheduler, and not reused anywhere else) then that's a reasonable trade-off. An ad-hoc list of individual issues that we know about and should be trying to solve (in no particular order) that all seem related to the data model design problem we are trying to take a stab at here: 1/ RequestSpec is an unversioned dict even though it's the central piece of a placement request for the scheduler Targeted by http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/request-spec-object.html 2/ There are scheduler_hints that are needed throughout the lifecycle of an instance but are never persisted so are lost after boot Targeted by http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/persist-request-spec.html 3/ We have the Migration objects that are used both for resource tracking for instances being migrated, and as an indication of an instance being moved, but are not used in all the places we need this kind of book keeping (live migration, rebuild) http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/robustify_evacuate.html is aiming to provide that evacuations, should we also propose to track rebuilds and live migrations as well ? 4/ Evacuate (an orchestrated rebuild) is especially problematic because it usually involves failure modes, which are difficult to identify and handle properly without a consistently used data model. Could you please describe more the problem ? Is it due to the fact that we're not persisting the request spec so it's basically a wet finger for knowing if that's good or not ? 5/ Some of the recently added constraints that influence resource tracking (NUMA, CPU pinning) cannot simply be calculated from the flavor on the fly when tracking resources, but need to be persisted after a successful claim as they are dependent on the state of the host at that very moment (see [1]) I just wonder if a scheduler API called 'check_destination(RequestSpec, destination) could help ? I mean, when live-migrating, we know that we try to check what we can, but since neither the spec is persisted, nor the migration is verifying that the destination host can fulfill the request, there are many bugs with that. FYI, I tried to propose that https://review.openstack.org/
Re: [openstack-dev] [nova] RequestSpec object and Instance model
On 06/02/2015 03:14 PM, Sylvain Bauza wrote: > Hi, > > Currently working on implementing the RequestSpec object BP [1], I had > some cool comments on my change here : > https://review.openstack.org/#/c/145528/12/nova/objects/request_spec.py,cm > > Since we didn't discussed on how to persist that RequestSpec object, I > think the comment is valuable. > > For the moment, the only agreed spec for persisting the object that we > have is [2] but there is also a corollar here that means that we would > have to persist more than the current fields > https://review.openstack.org/#/c/169901/3/specs/liberty/approved/add-buildrequest-obj.rst,cm > > > So, there are 2 possibilities : > #1, we only persist the RequestSpec for the sole Scheduler and in that > case, we can leave as it is - only a few fields from Instance are stored > #2, we consider that RequestSpec can be used for more than just the > Scheduler, and then we need to make sure that we will have all the > instance fields then. > So these are 2 possibilities if we agree that we need to make progress on the spec as is defined and merged now. What I was complaining yesterday is that we don't seem to have done enough of high level investigation into this stuff before embarking on writing a set of specs that then due to their format obscure the problems we are actually trying to solve. Work around the scheduler touches on a lot of issues that have only recently been noticed. While I am all for the incremental approach, it seems silly to completely disregard the issues we already know about. We should have a high level overview of the problems we know we want to solve, and then come up with an incremental way of solving them, but not without keeping an eye on the big picture at all times. An ad-hoc list of individual issues that we know about and should be trying to solve (in no particular order) that all seem related to the data model design problem we are trying to take a stab at here: 1/ RequestSpec is an unversioned dict even though it's the central piece of a placement request for the scheduler 2/ There are scheduler_hints that are needed throughout the lifecycle of an instance but are never persisted so are lost after boot 3/ We have the Migration objects that are used both for resource tracking for instances being migrated, and as an indication of an instance being moved, but are not used in all the places we need this kind of book keeping (live migration, rebuild) 4/ Evacuate (an orchestrated rebuild) is especially problematic because it usually involves failure modes, which are difficult to identify and handle properly without a consistently used data model. 5/ Some of the recently added constraints that influence resource tracking (NUMA, CPU pinning) cannot simply be calculated from the flavor on the fly when tracking resources, but need to be persisted after a successful claim as they are dependent on the state of the host at that very moment (see [1]) 6/ Related to the previous one - there is data related to the instance in addition to the flavor that need to follow the '_old' and '_new' pattern (needs the values related to both source and destination host persisted during a migration/resize/live migration/) 7/ The issues cells v2 folks are hitting (mentioned above) where they don't want to have any Instances in the top level cell but still need to persist stuff. 8/ Issues with having no access to individual instance UUIDs in the scheduler, but a lot of data access for more complex filtering revolves around it being present. Most of the above have individual bugs that I can try to find and link here too. [1] https://bugs.launchpad.net/nova/+bug/1417667 The overall theme of all the above is (to paraphrase alaski from IRC) how to organize the big blob of data that is an instance in all of it's possible states, in such a way that it makes sense, nothing is missing, there is as little duplication as possible, and access patterns of different services that require different bits can work without massive overhead. > > I'm not strongly opiniated on that, I maybe consider that #2 is probably > the best option but there is a tie in my mind. Help me figuring out > what's the best option. > If we want to keep things moving forward on this particular BP - I'd go with adding the RequestSpec object and make sure the code that uses it is migrated. I believe that spike alone will leave us with much better idea about the problem. In addition - writing a high level spec/wiki that we can refer back to in individual BPs and see how they solve it would be massively helpful too. N. > -Sylvain > > [1] : > http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/request-spec-object.html > > [2] : > http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/persist-request-spec.html > > > > __ > OpenStack Development Mailing List (not for usage question
[openstack-dev] [nova] RequestSpec object and Instance model
Hi, Currently working on implementing the RequestSpec object BP [1], I had some cool comments on my change here : https://review.openstack.org/#/c/145528/12/nova/objects/request_spec.py,cm Since we didn't discussed on how to persist that RequestSpec object, I think the comment is valuable. For the moment, the only agreed spec for persisting the object that we have is [2] but there is also a corollar here that means that we would have to persist more than the current fields https://review.openstack.org/#/c/169901/3/specs/liberty/approved/add-buildrequest-obj.rst,cm So, there are 2 possibilities : #1, we only persist the RequestSpec for the sole Scheduler and in that case, we can leave as it is - only a few fields from Instance are stored #2, we consider that RequestSpec can be used for more than just the Scheduler, and then we need to make sure that we will have all the instance fields then. I'm not strongly opiniated on that, I maybe consider that #2 is probably the best option but there is a tie in my mind. Help me figuring out what's the best option. -Sylvain [1] : http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/request-spec-object.html [2] : http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/persist-request-spec.html __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev