Re: [openstack-dev] [Nova][FFE] Feature freeze exception for virt-driver-numa-placement

[Murray, Paul (HP Cloud)]

On 4 September 2014 14:07, Nikola Đipanov ndipa...@redhat.com wrote:
On 09/04/2014 02:31 PM, Sean Dague wrote:
 On 09/04/2014 07:58 AM, Nikola Đipanov wrote:
 Hi team,

 I am requesting the exception for the feature from the subject (find
 specs at [1] and outstanding changes at [2]).

 Some reasons why we may want to grant it:

 First of all all patches have been approved in time and just lost the
 gate race.

 Rejecting it makes little sense really, as it has been commented on by a
 good chunk of the core team, most of the invasive stuff (db migrations
 for example) has already merged, and the few parts that may seem
 contentious have either been discussed and agreed upon [3], or can
 easily be addressed in subsequent bug fixes.

 It would be very beneficial to merge it so that we actually get real
 testing on the feature ASAP (scheduling features are not tested in the
 gate so we need to rely on downstream/3rd party/user testing for those).

 This statement bugs me. It seems kind of backwards to say we should
 merge a thing that we don't have a good upstream test plan on and put it
 in a release so that the testing will happen only in the downstream case.


The objective reality is that many other things have not had upstream
testing for a long time (anything that requires more than 1 compute node
in Nova for example, and any scheduling feature - as I mention clearly
above), so not sure how that is backwards from any reasonable point.

Thanks to folks using them, it is still kept working and bugs get fixed.
Getting features into the hands of users is extremely important...

 Anyway, not enough to -1 it, but enough to at least say something.


.. but I do not want to get into the discussion about software testing
here, not the place really.

However, I do think it is very harmful to respond to FFE request with
such blanket statements and generalizations, if only for the message it
sends to the contributors (that we really care more about upholding our
own myths as a community than users and features).


I believe you brought this up as one of your justifications for the FFE. When I 
read your statement it does sound as though you want to put experimental code 
in at the final release. I am sure that is not what you had in mind, but I am 
also sure you can also understand Sean's point of view. His point is clear and 
pertinent to your request.

As the person responsible for Nova in HP I will be interested to see how it 
operates in practice. I can assure you we will do extensive testing on it 
before it goes into the wild and we will not put it into practice if we are not 
happy.

Paul

Paul Murray
Nova Technical Lead, HP Cloud
+44 117 312 9309

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][FFE] Feature freeze exception for virt-driver-numa-placement

[Murray, Paul (HP Cloud)]

 Anyway, not enough to -1 it, but enough to at least say something.





 .. but I do not want to get into the discussion about software testing

 here, not the place really.



 However, I do think it is very harmful to respond to FFE request with

 such blanket statements and generalizations, if only for the message it

 sends to the contributors (that we really care more about upholding our

 own myths as a community than users and features).





 I believe you brought this up as one of your justifications for the FFE.
 When I read your statement it does sound as though you want to put
 experimental code in at the final release. I am sure that is not what
 you had in mind, but I am also sure you can also understand Sean's point
 of view. His point is clear and pertinent to your request.



 As the person responsible for Nova in HP I will be interested to see how
 it operates in practice. I can assure you we will do extensive testing
 on it before it goes into the wild and we will not put it into practice
 if we are not happy.


That is awesome and we as a project are lucky to have that! I would not
want things put into practice that users can't use or see huge flaws with.

I can't help but read this as you being OK with the feature going ahead,
though :).


Actually, let's say I have no particular objection. Just thought Sean's point 
is worth noting.

Now, if this had been done as an extensible resource I could easily decouple 
deploying it from all the bug fixes that come through with the release. But 
that's another matter...

Paul
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] [Gantt] Scheduler split status (updated)

[Murray, Paul (HP Cloud)]

Hi All,

I'm sorry I am so late to this lively discussion - it looks a good one! Jay has 
been driving the debate a bit so most of this is in response to his comments. 
But please, anyone should chip in.

On extensible resource tracking

Jay, I am surprised to hear you say no one has explained to you why there is an 
extensible resource tracking blueprint. It's simple, there was a succession of 
blueprints wanting to add data about this and that to the resource tracker and 
the scheduler and the database tables used to communicate. These included 
capabilities, all the stuff in the stats, rxtx_factor, the equivalent for cpu 
(which only works on one hypervisor I think), pci_stats and more were coming 
including,

https://blueprints.launchpad.net/nova/+spec/network-bandwidth-entitlement
https://blueprints.launchpad.net/nova/+spec/cpu-entitlement

So, in short, your claim that there are no operators asking for additional 
stuff is simply not true.

Around about the Icehouse summit (I think) it was suggested that we should stop 
the obvious trend and add a way to make resource tracking extensible, similar 
to metrics, which had just been added as an extensible way of collecting on 
going usage data (because that was also wanted).

The json blob you refer to was down to the bad experience of the 
compute_node_stats table implemented for stats - which had a particular 
performance hit because it required an expensive join. This was dealt with by 
removing the table and adding a string field to contain the data as a json 
blob. A pure performance optimization. Clearly there is no need to store things 
in this way and with Nova objects being introduced there is a means to provide 
strict type checking on the data even if it is stored as json blobs in the 
database.

On scheduler split

I have no particular position on splitting the scheduler. However, there was an 
interesting reaction to the network bandwidth entitlement blueprint listed 
above. The nova community felt it was a network thing and so nova should not 
provide it - neutron should. Of course, in nova, the nova scheduler makes 
placement decisions... can you see where this is going...? Nova needs to 
coordinate its placement decision with neutron to decide if a host has 
sufficient bandwidth available. Similar points are made about cinder - nova has 
no idea about cinder, but in some environments the location of a volume matters 
when you come to place an instance.

I should re-iterate that I have no position on splitting out the scheduler, but 
some way to deal with information from outside nova is certainly desirable. 
Maybe other services have the same dilemma.

On global resource tracker

I have to say I am inclined to be against the idea of turning the scheduler 
into a global resource tracker. I do see the benefit of obtaining a resource 
claim up front, we have all seen that the scheduler can make incorrect choices 
because of the delay in reflecting resource allocation to the database and so 
to the scheduler - it operates on imperfect information. However, it is best to 
avoid a global service relying on synchronous interaction with compute nodes 
during the process of servicing a request. I have looked at your example code 
for the scheduler (global resource tracker) and it seems to make a choice from 
local information and then interact with the chosen compute node to obtain a 
claim and then try again if the claim fails. I get it - I see that it deals 
with the same list of hosts on the retry. I also see it has no better chance of 
getting it right.

Your desire to have a claim is borne out by the persistent claims spec (I love 
the spec, I really I don't see why they have to be persistent). I think that is 
a great idea. Why not let the scheduler make placement suggestions (as a global 
service) and then allow conductors to obtain the claim and retry if the claim 
fails? Similar process to your code, but the scheduler only does its part and 
the conductors scale out the process by acting more locally and with more 
parallelism. (Of course, you could also be optimistic and allow the compute 
node to do the claim as part of the create as the degenerate case).

To emphasize the point further, what would a cells scheduler do? Would that 
also make a synchronous operation to obtain the claim?

My reaction to the global resource tracker idea has been quite negative. I want 
to like the idea because I like the thought of knowing I have the resources 
when I get my answer. Its just that I think the persistent claims (without the 
persistent part :) ) gives us a lot of what we need. But I am still open to be 
convinced.

Paul



On 07/14/2014 10:16 AM, Sylvain Bauza wrote:
 Le 12/07/2014 06:07, Jay Pipes a écrit :
 On 07/11/2014 07:14 AM, John Garbutt wrote:
 On 10 July 2014 16:59, Sylvain Bauza sbauza at redhat.com wrote:
 Le 10/07/2014 15:47, Russell Bryant a écrit :
 On 07/10/2014 05:06 AM, Sylvain Bauza wrote:
 Hi all,

 === tl;dr: Now that 

Re: [openstack-dev] [nova] Question about addit log in nova-compute.log

[Murray, Paul (HP Cloud)]

Comment inline at bottom of message...

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com] 
Sent: 06 May 2014 18:44
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [nova] Question about addit log in nova-compute.log

On 05/06/2014 01:37 PM, Jiang, Yunhong wrote:
 -Original Message-
 From: Jay Pipes [mailto:jaypi...@gmail.com]
 Sent: Monday, May 05, 2014 6:19 PM
 To: openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [nova] Question about addit log in 
 nova-compute.log

 On 05/05/2014 04:19 PM, Jiang, Yunhong wrote:
 -Original Message-
 From: Jay Pipes [mailto:jaypi...@gmail.com]
 Sent: Monday, May 05, 2014 9:50 AM
 To: openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [nova] Question about addit log in 
 nova-compute.log

 On 05/04/2014 11:09 PM, Chen CH Ji wrote:
 Hi
   I saw in my compute.log has following logs which
 looks
 to me strange at first, Free resource is negative make me confused
 and I
 take a look at the existing code
   looks to me the logic is correct and calculation
 doesn't
 have problem ,but the output 'Free' is confusing

   Is this on purpose or might need to be enhanced?

 2014-05-05 10:51:33.732 4992 AUDIT
 nova.compute.resource_tracker
 [-]
 Free ram (MB): -1559
 2014-05-05 10:51:33.732 4992 AUDIT
 nova.compute.resource_tracker
 [-]
 Free disk (GB): 29
 2014-05-05 10:51:33.732 4992 AUDIT
 nova.compute.resource_tracker
 [-]
 Free VCPUS: -3

 Hi Kevin,

 I think changing free to available might make things a little 
 more clear. In the above case, it may be that your compute worker 
 has both CPU and RAM overcommit enabled.

 Best,
 -jay

 HI, Jay,
 I don't think change 'free' to 'available' will make it clearer.
 IMHO, the calculation of the 'free' is bogus. When report the 
 status in
 the periodic task, the resource tracker has no idea of the 
 over-commit ration at all, thus it simply subtract the total RAM 
 number assigned to instances from the RAM number provided by 
 hypervisor w/o considering the over-commitment at all. So this number really 
 have meaningless.

 Agreed that in it's current state, it's meaningless. But... that 
 said, the numbers *could* be used to show oversubscription 
 percentage, and you don't need to know the max overcommit ratio in 
 order to calculate that with the numbers already known.

 I don't think user can use these number to calculate the 'available'. User 
 has to know the max overcommit ratio to know the 'available'. Also, it's 
 really ironic to provide some meaningless information and have the user to 
 calculate to get meaningful.

 This is related to https://bugs.launchpad.net/nova/+bug/1300775 . I think it 
 will be better if we can have the resource tracker to knows about the ratio.

Sorry, you misunderstood me... I was referring to the resource tracker above, 
not a regular user. The resource tracker already knows the total amount of 
physical resources available on each compute node, and it knows the resource 
usage reported by each compute node. Therefore, the resource tracker already 
has all the information it needs to understand the *actual* overcommit ratio of 
CPU and memory on each compute node, regardless of the settings of the 
*maximum* overcommit ratio on a compute node (which is in each compute node's 
nova.conf).

Hope that makes things a bit clearer! Sorry for the confusion :)

[pmurray]: In the past, when debugging problems in our systems, I have found it 
useful to know what the free value actually is. So I think a percentage/ratio 
value would be less helpful to me as an operator. Additionally, if the message 
changed depending on the sign of the value it would make it harder to search 
for it in the logs. So from that point of view I would say the message is fine 
as it is. I can see that free being negative seems odd. Could add (negative 
value means overcommitted resource) to the message?

Paul

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposal: Move CPU and memory allocation ratio out of scheduler

[Murray, Paul (HP Cloud)]

Hi Jay,

This sounds good to me. You left out the part of limits from the discussion – 
these filters set the limits used at the resource tracker. You also left out 
the force-to-host and its effect on limits. Yes, I would agree with doing this 
at the resource tracker too.

And of course the extensible resource tracker is the right way to do it ☺

Paul.

From: Jay Lau [mailto:jay.lau@gmail.com]
Sent: 04 June 2014 10:04
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova] Proposal: Move CPU and memory allocation 
ratio out of scheduler

Does there is any blueprint related to this? Thanks.

2014-06-03 21:29 GMT+08:00 Jay Pipes 
jaypi...@gmail.commailto:jaypi...@gmail.com:
Hi Stackers,

tl;dr
=

Move CPU and RAM allocation ratio definition out of the Nova scheduler and into 
the resource tracker. Remove the calculations for overcommit out of the 
core_filter and ram_filter scheduler pieces.

Details
===

Currently, in the Nova code base, the thing that controls whether or not the 
scheduler places an instance on a compute host that is already full (in terms 
of memory or vCPU usage) is a pair of configuration options* called 
cpu_allocation_ratio and ram_allocation_ratio.

These configuration options are defined in, respectively, 
nova/scheduler/filters/core_filter.py and nova/scheduler/filters/ram_filter.py.

Every time an instance is launched, the scheduler loops through a collection of 
host state structures that contain resource consumption figures for each 
compute node. For each compute host, the core_filter and ram_filter's 
host_passes() method is called. In the host_passes() method, the host's 
reported total amount of CPU or RAM is multiplied by this configuration option, 
and the product is then subtracted from the reported used amount of CPU or RAM. 
If the result is greater than or equal to the number of vCPUs needed by the 
instance being launched, True is returned and the host continues to be 
considered during scheduling decisions.

I propose we move the definition of the allocation ratios out of the scheduler 
entirely, as well as the calculation of the total amount of resources each 
compute node contains. The resource tracker is the most appropriate place to 
define these configuration options, as the resource tracker is what is 
responsible for keeping track of total and used resource amounts for all 
compute nodes.

Benefits:

 * Allocation ratios determine the amount of resources that a compute node 
advertises. The resource tracker is what determines the amount of resources 
that each compute node has, and how much of a particular type of resource have 
been used on a compute node. It therefore makes sense to put calculations and 
definition of allocation ratios where they naturally belong.
 * The scheduler currently needlessly re-calculates total resource amounts on 
every call to the scheduler. This isn't necessary. The total resource amounts 
don't change unless either a configuration option is changed on a compute node 
(or host aggregate), and this calculation can be done more efficiently once in 
the resource tracker.
 * Move more logic out of the scheduler
 * With the move to an extensible resource tracker, we can more easily evolve 
to defining all resource-related options in the same place (instead of in 
different filter files in the scheduler...)

Thoughts?

Best,
-jay

* Host aggregates may also have a separate allocation ratio that overrides any 
configuration setting that a particular host may have

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Thanks,
Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Thoughts about how to enforce DB and object models mapping

[Murray, Paul (HP Cloud)]

Hi Sylvain,

I think the idea is that anyone making changes to the db data model has to make 
changes to the corresponding object and that is captured in the data model 
implications of nova-specs (or it should be). Then reviewers check it is done. 
Tests on the objects will necessarily exercise the database fields by writing 
to and reading form them I think.

In the case of ComputeNode, it is still just getting there. I should be putting 
fields in as part of the resource-tracker-objects work track, and you will have 
seen by now that there are patches up for review doing that. Any help making it 
complete is welcome.

The philosophy is to only add fields that are being used - so as it is being 
built out, the existence of a field in db model does not necessarily imply it 
should be added to the ComputeNode object. It is easy to remove an unused field 
from the db, but more complicated to remove it from the object. Likewise for 
fields that want to change their format. So for my part, the fields that were 
not added yet are fields that were not used in the resource tracker - even if 
they are used in the scheduler.

The one exception is the pci_stats - which was undergoing change.

Paul.

-Original Message-
From: Sylvain Bauza [mailto:sba...@redhat.com] 
Sent: 06 June 2014 08:52
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [Nova] Thoughts about how to enforce DB and object 
models mapping

Hi,

By working on providing a new scheduler client for compute nodes, I began to 
use the ComputeNode object instead of placing a call to conductor directly.

Unfortunately, I recently discovered that some changes have been done in the DB 
model for ComputeNode that haven't been populated on the corresponding Object 
model.

As there is no current code in Nova using ComputeNode objects, I can understand 
that, as the first user of the object, I have to find the lacks and fix them, 
so I'm OK with that.

That said, I'm thinking about how we could make sure that any change in a DB 
model would have to be also done on the corresponding Object model.
A possible approach would be to create a test class for each object and provide 
a check against the DB model to make sure that at least the fields are all 
there.

Does that sound reasonable to you ? Should we provide another way for gating 
this ?

Thanks,
-Sylvain


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposal: Move CPU and memory allocation ratio out of scheduler

[Murray, Paul (HP Cloud)]

Forcing an instance to a specific host is very useful for the operator - it 
fulfills a valid use case for monitoring and testing purposes. I am not 
defending a particular way of doing this, just bringing up that it has to be 
handled. The effect on limits is purely implementation - no limits get set so 
it by-passes any resource constraints, which is deliberate.

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com] 
Sent: 04 June 2014 19:17
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [nova] Proposal: Move CPU and memory allocation 
ratio out of scheduler

On 06/04/2014 06:10 AM, Murray, Paul (HP Cloud) wrote:
 Hi Jay,

 This sounds good to me. You left out the part of limits from the 
 discussion - these filters set the limits used at the resource tracker.

Yes, and that is, IMO, bad design. Allocation ratios are the domain of the 
compute node and the resource tracker. Not the scheduler. The allocation ratios 
simply adjust the amount of resources that the compute node advertises to 
others. Allocation ratios are *not* scheduler policy, and they aren't related 
to flavours.

 You also left out the force-to-host and its effect on limits.

force-to-host is definitively non-cloudy. It was a bad idea that should never 
have been added to Nova in the first place.

That said, I don't see how force-to-host has any affect on limits. 
Limits should not be output from the scheduler. In fact, they shouldn't be 
anything other than an *input* to the scheduler, provided in each host state 
struct that gets built from records updated in the resource tracker and the 
Nova database.

  Yes, I
 would agree with doing this at the resource tracker too.

 And of course the extensible resource tracker is the right way to do 
 it J

:) Yes, clearly this is something that I ran into while brainstorming around 
the extensible resource tracker patches.

Best,
-jay

 Paul.

 *From:*Jay Lau [mailto:jay.lau@gmail.com]
 *Sent:* 04 June 2014 10:04
 *To:* OpenStack Development Mailing List (not for usage questions)
 *Subject:* Re: [openstack-dev] [nova] Proposal: Move CPU and memory 
 allocation ratio out of scheduler

 Does there is any blueprint related to this? Thanks.

 2014-06-03 21:29 GMT+08:00 Jay Pipes jaypi...@gmail.com
 mailto:jaypi...@gmail.com:

 Hi Stackers,

 tl;dr
 =

 Move CPU and RAM allocation ratio definition out of the Nova scheduler 
 and into the resource tracker. Remove the calculations for overcommit 
 out of the core_filter and ram_filter scheduler pieces.

 Details
 ===

 Currently, in the Nova code base, the thing that controls whether or 
 not the scheduler places an instance on a compute host that is already 
 full (in terms of memory or vCPU usage) is a pair of configuration
 options* called cpu_allocation_ratio and ram_allocation_ratio.

 These configuration options are defined in, respectively, 
 nova/scheduler/filters/core_filter.py and 
 nova/scheduler/filters/ram_filter.py.

 Every time an instance is launched, the scheduler loops through a 
 collection of host state structures that contain resource consumption 
 figures for each compute node. For each compute host, the core_filter 
 and ram_filter's host_passes() method is called. In the host_passes() 
 method, the host's reported total amount of CPU or RAM is multiplied 
 by this configuration option, and the product is then subtracted from 
 the reported used amount of CPU or RAM. If the result is greater than 
 or equal to the number of vCPUs needed by the instance being launched, 
 True is returned and the host continues to be considered during 
 scheduling decisions.

 I propose we move the definition of the allocation ratios out of the 
 scheduler entirely, as well as the calculation of the total amount of 
 resources each compute node contains. The resource tracker is the most 
 appropriate place to define these configuration options, as the 
 resource tracker is what is responsible for keeping track of total and 
 used resource amounts for all compute nodes.

 Benefits:

   * Allocation ratios determine the amount of resources that a compute 
 node advertises. The resource tracker is what determines the amount of 
 resources that each compute node has, and how much of a particular 
 type of resource have been used on a compute node. It therefore makes 
 sense to put calculations and definition of allocation ratios where 
 they naturally belong.
   * The scheduler currently needlessly re-calculates total resource 
 amounts on every call to the scheduler. This isn't necessary. The 
 total resource amounts don't change unless either a configuration 
 option is changed on a compute node (or host aggregate), and this 
 calculation can be done more efficiently once in the resource tracker.
   * Move more logic out of the scheduler
   * With the move to an extensible resource tracker, we can more 
 easily evolve to defining all resource-related options in the same 
 place (instead

[openstack-dev] [Nova] structure of supported_instances field (architecture, hypervisor type, vm mode) ?

[Murray, Paul (HP Cloud)]

Hi All,

I have a question about information used to determine if a host supports an 
image.

The virt drivers all provide a list of triples of the form (architecture, 
hypervisor type, vm mode). Each triple is compared to the corresponding three 
image properties in a request in the scheduler filter called 
image_props_filter.py to determine if a host can support the given image. This 
information is held in a field in the compute_nodes table called 
supported_instances.

I am adding the supported_instances field to the ComputeNode object. As it has 
a definite structure I want to make sure the structure is checked.

So here is my question:

Am I right in thinking that there are always three properties in each element 
in the list (architecture, hypervisor type, vm mode)?

As far as I can tell, these are the only values checked in the filter and all 
the virt drivers always generate those three. If so, is it reasonable for me to 
insist the elements of the supported_instance list have three strings in them 
when the object checks the data type? I don't want to restrict the structure to 
being strictly a list of triples if there might in fact be a variable number of 
properties provided.

Paul
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] structure of supported_instances field (architecture, hypervisor type, vm mode) ?

[Murray, Paul (HP Cloud)]

FYI I'm currently working on changing the get_available_resources() method
to return a formal object, instead of its horrible undocumented dict().
With this, the current list of lists / JSON used for supported_instances
will turn into a formal object with strict validation.

Regards,
Thanks Daniel, we had better coordinate on that then. I will try and catch you 
on IRC today.

Paul
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] object field naming question

[Murray, Paul (HP Cloud)]

Hi All,

The following question relates to this change:

https://review.openstack.org/#/c/125091/

This change adds a field to the ComputeNode object called 
supported_instances. It also adds an object called SupportedInstance that 
has fields called arch, hvtype and vm_mode.

All these names already exist in the nova code, but when put together in these 
objects they seem a little odd (i.e. supported_instances may be a little 
misleading, hvtype has no hyphen but vm_mode does). This is where they come 
from:


-  supported_instances is the name of the corresponding field of the 
compute_nodes database table. The supported_instances field actually contains a 
the list of architecture, hypervisor type and vm_mode combinations supported by 
the compute node. It is also the existing field name used in a dict provided by 
the virt drivers to report this list to nova.


-  arch, hvtype and vm_mode are the names used by variables throughout 
nova that refer to the relevant data obtained contained in supported_instances.

The question is: are these the names we actually want to use?

The reason I ask is that once the names are used for Nova object fields, the 
nova object versions would need to be bumped to change them. So if they are 
going to change, now would be a good time to put the right thing in the objects 
(the rest of nova could be changed subsequently).

So, do you think supported_instances should be something else, e.g. 
supported_hv_properties? (The SupportedInstance object name can follow this 
one.) Please make a suggestion if you think it should change.

Do you think we should have hyphens or not in hvtype and vm_mode (note, vm_mode 
occurs 160+ times in nova and vmmode occurs 12 times, whereas occurrence of 
hv_type vs hvtype has the opposite bias).

In fact, is there a convention that should be followed that I have forgotten 
about (or never knew)?

Let me know opinions and I will fix the patch accordingly.

Thanks,
Paul
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] object field naming question

[Murray, Paul (HP Cloud)]

I think the hyphen question is settled – we can go with hv_type and vm_mode to 
be in line with the rest of nova.

Now for the other bit…

After this email thread and discussion in the IRC with those who expressed 
interest I have come to the following:

I will use the name supported_hv_specs instead of supported_instances in the 
ComputeNode object and HVSpec for the new object. So the field type of 
supported_hv_specs will be ListOfObjectsField(‘HVSpec’).

The supported_instances field in the compute_nodes table will be left as it is. 
This means that supported_hv_specs in ComputeNodes will map to the 
supported_instances field in the compute_nodes table.

Thanks for helping,
Paul




From: Paul Murray [mailto:ptma...@gmail.com]
Sent: 10 October 2014 13:00
To: Murray, Paul (HP Cloud)
Subject: Fwd: [openstack-dev] [Nova] object field naming question


-- Forwarded message --
From: Dan Smith d...@danplanet.commailto:d...@danplanet.com
Date: 9 October 2014 17:40
Subject: Re: [openstack-dev] [Nova] object field naming question
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org


 The value it adds (and that an underscore would add in hvtype -
 hv_type) is that the name would match the naming style for the vast
 majority of everything else in OpenStack. See, for examples:

Agreed.

 As mentioned in the review, I disagree on this point, since doing a
 cleanup afterwards would mean having to increment the
 nova.objects.SupportedInstance model VERSION as soon as it went into
 master. I say let's make the quick change now and avoid having to write
 code like this in the next patchset:

  if client_version = (1,0):
 # We renamed hvtype to hv_type in 1.1
 self.hv_type = fields.get('hvtype')

Right, this becomes RPC debt if we think we might change it later. We
definitely want to get it right the first time whenever possible.

--Dan


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Pulling nova/virt/hardware.py into nova/objects/

[Murray, Paul (HP Cloud)]

I spent a little time trying to work out a good way to include this kind of 
data in the ComputeNode object. You will have seen that I added the 
supported_instances reported to the RT by the virt drivers as a list of HVSpec 
– where HVSpec is a new nova object I created for the purpose.

 The rationale behind two parallel data model hiercharies is that the
 format the virt drivers report data in, is not likely to be exactly
 the same as the format that the resoure tracker / scheduler wishes to
 use in the database.

Yeah, and in cases where we know where that line is, it makes sense to
use the lighter-weight modeling for sure.

Something that happens in some cases in RT is the data reported by the virt 
driver is modified by the RT – this is certainly the case for stats (which 
should probably not have been used by the virt driver – ironic in this case). 
In other cases memory, cpu, disk etc are split out into other fields at the 
moment (which is where Jay Pipes is working on a model for resources in 
general).

Where the data in an object field is not simple (e.g. lists of something) it is 
not really easy to work with in an object. You can’t just add to a list that is 
already stored in an object field, you need to make sure the object knows it 
has been updated. So the easiest way to work is to add entire fields to the 
object and not change them in situ. So that seems to me like dealing with 
non-nova object data types makes sense as something separate to, say, the 
ComputeNode object.

An alternative would be to work on the way nova object handle nested data 
structures (i.e. recognizing updates in nested structures, api allowing for 
list manipulation etc.) It depends whether you think the objects are just doing 
versioning to support upgrade/mixed versions or are a general purpose object 
model.

Note that this happens with NUMA data structures and PCI as well at the moment.

 FWIW, my patch series is logically split up into two parts. THe first
 10 or so patches are just thought of as general cleanup and useful to
 Nova regardless of what we decide todo. The second 10 or so patches
 are where the objects start appearing  getting used  the controversial
 bits needing mor detailed discussion.

Right, so after some discussion I think we should go ahead and merge the
bottom of this set (all of them are now acked I think) and continue the
discussion on the top half where the modeling is introduced.


Paul
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gantt] Scheduler sub-group meeting agenda 11/11

[Murray, Paul (HP Cloud)]

The resource tracker objects BP was grouped under the scheduler work items as 
well:

http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/make-resource-tracker-use-objects.html



-

1)  Summit recap

2)  Status of BPs:

a.   Isolate scheduler DB aggregates - 
https://review.openstack.org/#/c/89893/

b.  Isolate scheduler DB for instance groups - 
https://review.openstack.org/#/c/131553/

c.   Detach service from compute node - 
https://review.openstack.org/#/c/126895/

d.  Model resource objects - https://review.openstack.org/#/c/127609/

e.  Model request spec object - https://review.openstack.org/#/c/127610/

f.Change select_destination() to use RequestSpec object - 
https://review.openstack.org/#/c/127612/

g.   Convert virt/driver.py get_available_resources - 
https://blueprints.launchpad.net/nova/+spec/virt-driver-get-available-resources-object


--
Don Dugger
Censeo Toto nos in Kansa esse decisse. - D. Gale
Ph: 303/443-3786tel:303%2F443-3786


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic][nova] ironic driver retries on ironic driver Conflict response

[Murray, Paul (HP Cloud)]

Hi All,

Looking at the ironic virt driver code in nova it seems that a Conflict (409) 
response from the ironic client results in the driver re-trying the request. 
Given the comment below in the ironic code I would imagine that is not the 
right behavior - it reads as though this is something that would fail on the 
retry as well.

class Conflict(HTTPClientError):
HTTP 409 - Conflict.

Indicates that the request could not be processed because of conflict
in the request, such as an edit conflict.

http_status = 409
message = _(Conflict)

An example of this is if the virt driver attempts to assign an instance to a 
node that is in the power on state it will issue this Conflict response.

Have I understood this or is there something about this I am not getting right?

Paul


Paul Murray
Nova Technical Lead, HP Cloud
+44 117 316 2527

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Networks are not cleaned up in build failure

[Murray, Paul (HP Cloud)]

Hi All,

I recently experienced failures getting images from Glance while spawning 
instances. This step comes after building the networks in the guild sequence. 
When the Glance failure occurred the instance was cleaned up and rescheduled as 
expected, but the networks were not cleaned up. On investigation I found that 
the cleanup code for the networks is in the compute manager's 
_do_build_run_instance() method as follows:


# NOTE(comstud): Deallocate networks if the driver wants
# us to do so.
if self.driver.deallocate_networks_on_reschedule(instance):
self._cleanup_allocated_networks(context, instance,
requested_networks)

The default behavior in for the deallocate_networks_on_schedule() method 
defined in ComputeDriver is:

def deallocate_networks_on_reschedule(self, instance):
Does the driver want networks deallocated on reschedule?
return False

Only the Ironic driver over rides this method to return True, so I think this 
means the networks will not be cleaned up for any other virt driver.



Is this really the desired behavior?



I have filed a bug for this and plan to fix it: 
https://bugs.launchpad.net/nova/+bug/1410739



My initial thought is to fix this either by making the method in the base class 
return True or by adding the method to virt drivers returning True (I would 
expect the former). But I wanted to check if there is a reason for the base 
class behavior (and so the default behavior) to be *NOT* to clean up the 
networks?



Paul


Paul Murray
Nova Technical Lead, HP Cloud
+44 117 316 2527

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Ironic] Question about scheduling two instances to same baremetal node

[Murray, Paul (HP Cloud)]

There is bug when running nova with ironic 
https://bugs.launchpad.net/nova/+bug/1402658

I filed this bug – it has been a problem for us.

The problem is at scheduler side the IronicHostManager will consume all the 
resources for that node whatever
how much resource the instance used. But at compute node side, the 
ResourceTracker won't consume resources
like that, just consume like normal virtual instance. And ResourceTracker will 
update the resource usage once the
instance resource claimed, then scheduler will know there are some free 
resource on that node, then will try to
schedule other new instance to that node

You have summed up the problem nicely – i.e.: the resource availability is 
calculated incorrectly for ironic nodes.

I take look at that, there is NumInstanceFilter, it will limit how many 
instance can schedule to one host. So can
we just use this filter to finish the goal? The max instance is configured by 
option 'max_instances_per_host', we
can make the virt driver to report how many instances it supported. The ironic 
driver can just report max_instances_per_host=1.
And libvirt driver can report max_instance_per_host=-1, that means no limit. 
And then we can just remove the
IronicHostManager, then make the scheduler side is more simpler. Does make 
sense? or there are more trap?


Makes sense, but solves the wrong problem. The problem is what you said above – 
i.e.: the resource availability is calculated incorrectly for ironic nodes.
The right solution would be to fix the resource tracker. The ram resource on an 
ironic node has different allocation behavior to a regular node. The test to 
see if a new instance fits is the same, but instead of deducting the requested 
amount to get the remaining availability it should simply return 0. This should 
be dealt with in the new resource objects ([2] below) by either having 
different version of the resource object for ironic nodes (certainly doable and 
the most sensible option – resources should be presented according to the 
resources on the host). Alternatively the ram resource object should cater for 
the difference in its calculations.
I have a local fix for this that I was too shy to propose upstream because it’s 
a bit hacky and will hopefully be obsolete soon. I could share it if you like.
Paul
[2] https://review.openstack.org/#/c/127609/


From: Sylvain Bauza sba...@redhat.commailto:sba...@redhat.com
Date: 9 January 2015 at 09:17
Subject: Re: [openstack-dev] [Nova][Ironic] Question about scheduling two 
instances to same baremetal node
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org


Le 09/01/2015 09:01, Alex Xu a écrit :
Hi, All

There is bug when running nova with ironic 
https://bugs.launchpad.net/nova/+bug/1402658

The case is simple: one baremetal node with 1024MB ram, then boot two instances 
with 512MB ram flavor.
Those two instances will be scheduling to same baremetal node.

The problem is at scheduler side the IronicHostManager will consume all the 
resources for that node whatever
how much resource the instance used. But at compute node side, the 
ResourceTracker won't consume resources
like that, just consume like normal virtual instance. And ResourceTracker will 
update the resource usage once the
instance resource claimed, then scheduler will know there are some free 
resource on that node, then will try to
schedule other new instance to that node.

I take look at that, there is NumInstanceFilter, it will limit how many 
instance can schedule to one host. So can
we just use this filter to finish the goal? The max instance is configured by 
option 'max_instances_per_host', we
can make the virt driver to report how many instances it supported. The ironic 
driver can just report max_instances_per_host=1.
And libvirt driver can report max_instance_per_host=-1, that means no limit. 
And then we can just remove the
IronicHostManager, then make the scheduler side is more simpler. Does make 
sense? or there are more trap?

Thanks in advance for any feedback and suggestion.


Mmm, I think I disagree with your proposal. Let me explain by the best I can 
why :

tl;dr: Any proposal unless claiming at the scheduler level tends to be wrong

The ResourceTracker should be only a module for providing stats about compute 
nodes to the Scheduler.
How the Scheduler is consuming these resources for making a decision should 
only be a Scheduler thing.

Here, the problem is that the decision making is also shared with the 
ResourceTracker because of the claiming system managed by the context manager 
when booting an instance. It means that we have 2 distinct decision makers for 
validating a resource.

Let's stop to be realistic for a moment and discuss about what could mean a 
decision for something else than a compute node. Ok, let say a volume.
Provided that *something* would report the volume statistics to the Scheduler, 
that would be the 

Re: [openstack-dev] [nova] Deprecation of ComputeFilter

[Murray, Paul (HP Cloud)]

Hi Sylvain,

The list of filters does not only determine what conditions are checked, it 
also specifies the order in which they are checked.

If I read the code right this change creates the worst case efficiency for this 
filter. Normally you would filter first on something that removes as many nodes 
as possible to cut down the list. It is not normal for large numbers of hosts 
to be disabled, so this filter should normally come low down the list. This 
change effectively puts it at the top. 

As an example, imagine you are using AvailabilityZoneFilter and ComputeFilter. 
Let's say you have three AZs and at any one time a small percentage of your 
nodes are disabled. These are realistic circumstances. In this case you would 
filter on the AvaiabilityZoneFilter first and ComputeFilter last. The AZ will 
cut the number of hosts being considered by two thirds with the ComputeFilter 
only being executed against the remaining third. If the order is reversed both 
filters are run against almost all nodes.

Note he following:
1: the default driver for the servicegroup api is db, so this adds a db lookup 
for every node that would otherwise only be called for remaining nodes after 
executing other filters.
2: if the driver uses a local in memory cache this is not so bad - but that's 
not the default

Even if this filter seems dumb, it is still a filtering operation, so why not 
leave it as a filter in the same model as all the others and under the 
operators control?

Paul

-Original Message-
From: Sylvain Bauza [mailto:sba...@redhat.com] 
Sent: 06 March 2015 15:19
To: OpenStack Development Mailing List (not for usage questions)
Cc: openstack-operat...@lists.openstack.org
Subject: [openstack-dev] [nova] Deprecation of ComputeFilter

Hi,

First, sorry for cross-posting on both dev and operator MLs but I also would 
like to get operators feedback.

So, I was reviewing the scheduler ComputeFilter and I was wondering why the 
logic should be in a filter.
We indeed already have a check on the service information each time that a 
request is coming in, which is done by
HostManager.get_all_host_states() - basically called by
FilterScheduler._schedule()

Instead, I think it is error-prone to leave that logic in a filter because it 
can easily be accidentally removed from the list of filters. 
Besides, the semantics of the filter is not well known and operators could not 
understand that it is filtering on a Service RPC status, not the real compute 
node behind it.

In order to keep a possibility for operators to explicitely ask the 
FilterScheduler to also filter on disabled hosts, I propose to add a config 
option which would be self-explicit.

So, I made a quick POC for showing how we could move the logic to HostManager 
[1]. Feel free to review it and share your thoughts both on the change and 
here, because I want to make sure that we get a consensus on the removal before 
really submitting anything.


[1] https://review.openstack.org/#/c/162180/

-Sylvain


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] pci stats format and functional tests

[Murray, Paul (HP Cloud)]

Hi All,

I know Yunhong Jiang and Daniel Berrange have been involved in the following, 
but I thought it worth sending to the list for visibility.

While writing code to convert the resource tracker to use the ComputeNode 
object realized that the api samples used in the functional tests are not the 
same as the format as the PciDevicePool object. For example: 
hypervisor-pci-detail-resp.json has something like this:

os-pci:pci_stats: [
{
count: 5,
extra_info: {
key1: value1,
phys_function: [[\0x\, \0x04\, \0x00\, \0x1\]]
},
keya: valuea,
product_id: 1520,
vendor_id: 8086
}
],

My understanding from interactions with yjiang5 in the past leads me to think 
that something like this is what is actually expected:

os-pci:pci_stats: [
{
count: 5,
key1: value1,
phys_function: [[\0x\, \0x04\, \0x00\, \0x1\]],
keya: valuea,
product_id: 1520,
vendor_id: 8086
}
],

This is the way the PciDevicePool object expects the data structure to be and 
is also the way the libvirt virt driver creates pci device information (i.e. 
without the extra_info key). Other than that (which is actually pretty clear) 
I couldn't find anything to tell me definitively if my interpretation is 
correct and I don't want to change the functional tests without being sure they 
are wrong. So if anyone can give some guidance here I would appreciate it.

I separated this stuff out into a patch with a couple of other minor cleanups 
in preparation for the ComputeNode change, see: 
https://review.openstack.org/#/c/161843

Let me know if I am on the right track,

Cheers,
Paul


Paul Murray
Nova Technical Lead, HP Cloud
+44 117 316 2527

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Would anyone from Nova Core be able to join the Cinder mid-cycle meetup?

[Murray, Paul (HP Cloud)]

Hi jay,

There are a few Nova people in HP who intend to work on this area (perhaps you 
have heard that from Scott or Duncan). In fact I brought up that this would be 
coming up in Vancouver and that we intend to support it. Unfortunately none of 
us are available that week. I would like to be kept involved and will have to 
catch up with the other HP guys after the event.

In the meantime, if you want names to be involved beyond the cinder mid-cycle 
you can put down as a place holder for HP and I will sort out people to be 
involved when we are all back from vacation.

Paul

-Original Message-
From: Jay S. Bryant [mailto:jsbry...@electronicjungle.net] 
Sent: 24 July 2015 22:50
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [nova][cinder] Would anyone from Nova Core be able to 
join the Cinder mid-cycle meetup?

All,

I had the opportunity to chat with John Garbutt when he was here in Rochester 
for the Nova mid-cycle meet-up.  We discussed the fact that there was much to 
be gained by improving the communication between the Cinder and Nova teams.

With that idea in mind, it was suggested that the Cinder team open an 
invitation to Nova members to attend our mid-cycle meet-up.  The mid-cycle 
meet-up, of course, is not a secret.  A member of the Nova team has always been 
welcome to join, just hoping that an explicit invitation here may spark some 
interest.  :-)  Note:  John considered attending but is unable to do so.

The mid-cycle meet-up for Cinder is 8/4 through 8/7 at the HP site in Fort 
Collins, CO .  Friday is an optional code sprint day for those who are able to 
stay that long.  Details about the meet-up can be seen on our mid-cycle meetup 
planning etherpad [1].

If you are able to join us and help us work through the various challenges we 
are having between Cinder and Nova it would be greatly appreciated!

Jay

[1] https://etherpad.openstack.org/p/cinder-liberty-midcycle-meetup

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Would anyone from Nova Core be able to join the Cinder mid-cycle meetup?

[Murray, Paul (HP Cloud)]

 In the meantime, if you want names to be involved beyond the cinder mid-cycle 
 you can put down as a place holder for HP and I will sort out people to be 
 involved when we are all back from vacation.

Of course I meant you can put ME down :)

Paul

-Original Message-
From: Murray, Paul (HP Cloud) 
Sent: 28 July 2015 12:52
To: 'jsbry...@electronicjungle.net'; OpenStack Development Mailing List (not 
for usage questions)
Cc: Rosa, Andrea (HP Cloud Services)
Subject: RE: [openstack-dev] [nova][cinder] Would anyone from Nova Core be able 
to join the Cinder mid-cycle meetup?

Hi jay,

There are a few Nova people in HP who intend to work on this area (perhaps you 
have heard that from Scott or Duncan). In fact I brought up that this would be 
coming up in Vancouver and that we intend to support it. Unfortunately none of 
us are available that week. I would like to be kept involved and will have to 
catch up with the other HP guys after the event.

In the meantime, if you want names to be involved beyond the cinder mid-cycle 
you can put down as a place holder for HP and I will sort out people to be 
involved when we are all back from vacation.

Paul

-Original Message-
From: Jay S. Bryant [mailto:jsbry...@electronicjungle.net] 
Sent: 24 July 2015 22:50
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [nova][cinder] Would anyone from Nova Core be able to 
join the Cinder mid-cycle meetup?

All,

I had the opportunity to chat with John Garbutt when he was here in Rochester 
for the Nova mid-cycle meet-up.  We discussed the fact that there was much to 
be gained by improving the communication between the Cinder and Nova teams.

With that idea in mind, it was suggested that the Cinder team open an 
invitation to Nova members to attend our mid-cycle meet-up.  The mid-cycle 
meet-up, of course, is not a secret.  A member of the Nova team has always been 
welcome to join, just hoping that an explicit invitation here may spark some 
interest.  :-)  Note:  John considered attending but is unable to do so.

The mid-cycle meet-up for Cinder is 8/4 through 8/7 at the HP site in Fort 
Collins, CO .  Friday is an optional code sprint day for those who are able to 
stay that long.  Details about the meet-up can be seen on our mid-cycle meetup 
planning etherpad [1].

If you are able to join us and help us work through the various challenges we 
are having between Cinder and Nova it would be greatly appreciated!

Jay

[1] https://etherpad.openstack.org/p/cinder-liberty-midcycle-meetup

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] About rebuilding volume-backed instances.

[Murray, Paul (HP Cloud)]

> Unfortunately, you're trying to work around misuse of the cloud API's, not
> missing features from them. Don't use those volume types, and don't build
> systems that rely on single ports and interfaces. IMO rebuild is a misguided
> concept (something that took me a long time to realize). 

Slightly tangential to this discussion but specifically to your comment about
rebuild: HP (and I believe RAX) kept asking OpenStack infra to use rebuild in 
node pool instead of constantly deleting and creating instances.
The reason being it would give them a significant performance advantage
and dramatically improve the use of resources in the operators cloud. Node
pool would have gained a vastly better resource utilization. The
fact infra did not do that (I believe) is partly because it was difficult to 
refactor
node pool for the purpose (after all, it works and there's other things to do) 
and
partly because the resources were free. In a pay-per-use scenario the decision
would have been different.

This is off topic because it's not about cinder volumes. But I guess there are 
two
take aways: 

On one hand, in the end they didn't really need rebuild - node pool works 
without it.

On the other, it would have reduced their costs (if paying) and made better use 
of
resources.

I'll leave it to the reader to decide which of these matter to you.

> It requires one
> service (Nova) to take all of the responsibility for cross-service 
> reinitialization
> and that doesn't happen so you get weird disconnects like this one. Heat
> would be a better choice, as you can simply deploy a new template, which
> will replace the old instance in a relatively safe way including detaching and
> re-attaching volumes and any VIP's.
> 
> So, to be clear:
> 
> *DO* build systems that store data on volumes that are _not_ the root disk.
> Boot a new instance, initialize the configuration using your tools, and then
> move the volume attachment from the old to the new.
> 
> *DO* build systems that use DNS or a VIP to communicate so that new ports
> can be allocated and attached to the new instance while the old one is stil
> active.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] live migration sub-team meeting

[Murray, Paul (HP Cloud)]

So, to confirm:

The live migration IRC meeting will be held on: Tuesdays at 1400 UTC in 
#openstack-meeting-3, starting next week: Tuesday 17th November.

See: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

On that page you will find links the tracking pages and bugs. Please check that 
the appropriate specs and code reviews are listed on the tracking pages. Please 
also take time to review specs and code.

Please contact me or respond to this email thread if there is anything you want 
to see on the agenda for the first meeting.

Regards,
Paul

From: Murray, Paul (HP Cloud)
Sent: 10 November 2015 11:48
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] live migration sub-team meeting

Thank you for the prompting Michael. I was chasing up a couple of key people to 
make sure they were available.

The IRC meeting should be Tuesdays at 1400 UTC on #openstack-meeting-3 starting 
next week (too late for today).

I will get that sorted out with infra and send another email to confirm. I will 
also sort out all the specs and patches that I know about today. More 
information will be included about that too.

Paul

From: Michael Still [mailto:mi...@stillhq.com]
Sent: 09 November 2015 21:34
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] live migration sub-team meeting

So, its been a week. What time are we picking?

Michael

On Thu, Nov 5, 2015 at 10:46 PM, Murray, Paul (HP Cloud) 
<pmur...@hpe.com<mailto:pmur...@hpe.com>> wrote:
> > Most team members expressed they would like a regular IRC meeting for
> > tracking work and raising blocking issues. Looking at the contributors
> > here [2], most of the participants seem to be in the European
> > continent (in time zones ranging from UTC to UTC+3) with a few in the
> > US (please correct me if I am wrong). That suggests that a time around
> > 1500 UTC makes sense.
> >
> > I would like to invite suggestions for a day and time for a weekly
> > meeting -
>
> Maybe you could create a quick Doodle poll to reach a rough consensus on
> day/time:
>
> http://doodle.com/

Yes, of course, here's the poll:

http://doodle.com/poll/rbta6n3qsrzcqfbn



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Rackspace Australia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Nova live migration subteam meeting

[Murray, Paul (HP Cloud)]

A quick reminder: the first live migration subteam IRC meeting is today at 1400 
UTC on #openstack-meeting-3.

See: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Regards,
Paul

Paul Murray
Nova Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] live migration sub-team meeting

[Murray, Paul (HP Cloud)]

Thank you for the prompting Michael. I was chasing up a couple of key people to 
make sure they were available.

The IRC meeting should be Tuesdays at 1400 UTC on #openstack-meeting-3 starting 
next week (too late for today).

I will get that sorted out with infra and send another email to confirm. I will 
also sort out all the specs and patches that I know about today. More 
information will be included about that too.

Paul

From: Michael Still [mailto:mi...@stillhq.com]
Sent: 09 November 2015 21:34
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] live migration sub-team meeting

So, its been a week. What time are we picking?

Michael

On Thu, Nov 5, 2015 at 10:46 PM, Murray, Paul (HP Cloud) 
<pmur...@hpe.com<mailto:pmur...@hpe.com>> wrote:
> > Most team members expressed they would like a regular IRC meeting for
> > tracking work and raising blocking issues. Looking at the contributors
> > here [2], most of the participants seem to be in the European
> > continent (in time zones ranging from UTC to UTC+3) with a few in the
> > US (please correct me if I am wrong). That suggests that a time around
> > 1500 UTC makes sense.
> >
> > I would like to invite suggestions for a day and time for a weekly
> > meeting -
>
> Maybe you could create a quick Doodle poll to reach a rough consensus on
> day/time:
>
> http://doodle.com/

Yes, of course, here's the poll:

http://doodle.com/poll/rbta6n3qsrzcqfbn



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Rackspace Australia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] live migration sub-team meeting

[Murray, Paul (HP Cloud)]

Hi all,

Live migration was confirmed as a Nova priority for the Mitaka cycle and a 
sub-team section can be found on the priorities tracking page [1].

Most team members expressed they would like a regular IRC meeting for tracking 
work and raising blocking issues. Looking at the contributors here [2], most of 
the participants seem to be in the European continent (in time zones ranging 
from UTC to UTC+3) with a few in the US (please correct me if I am wrong). That 
suggests that a time around 1500 UTC makes sense.

I would like to invite suggestions for a day and time for a weekly meeting - 
you do not have to go by the above suggestion. When we have a time I will 
arrange the meeting and create a corresponding wiki page.

Please note that attendance is not a requirement for participation, but it will 
help to track or coordinate some activities.

Paul

[1]  https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking
[2] https://etherpad.openstack.org/p/mitaka-live-migration

Paul Murray
Nova Technical Lead, HP Cloud
+44 117 316 2527

Hewlett-Packard Limited   |   Registered Office: Cain Road, Bracknell, 
Berkshire, RG12 1HN   |Registered No: 690597 England   |VAT Number: GB 
314 1496 79

This e-mail may contain confidential and/or legally privileged material for the 
sole use of the intended recipient.  If you are not the intended recipient (or 
authorized to receive for the recipient) please contact the sender by reply 
e-mail and delete all copies of this message.  If you are receiving this 
message internally within the Hewlett Packard group of companies, you should 
consider the contents "CONFIDENTIAL".

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] live migration sub-team meeting

[Murray, Paul (HP Cloud)]

> > Most team members expressed they would like a regular IRC meeting for
> > tracking work and raising blocking issues. Looking at the contributors
> > here [2], most of the participants seem to be in the European
> > continent (in time zones ranging from UTC to UTC+3) with a few in the
> > US (please correct me if I am wrong). That suggests that a time around
> > 1500 UTC makes sense.
> >
> > I would like to invite suggestions for a day and time for a weekly
> > meeting -
> 
> Maybe you could create a quick Doodle poll to reach a rough consensus on
> day/time:
> 
> http://doodle.com/

Yes, of course, here's the poll: 

http://doodle.com/poll/rbta6n3qsrzcqfbn 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] attaching and detaching volumes in the API

[Murray, Paul (HP Cloud)]

Normally operations on instances are synchronized at the compute node. In some 
cases it is necessary to synchronize somehow at the API. I have one of those 
cases and wondered what is a good way to go about it.

As part of this spec: https://review.openstack.org/#/c/221732/

I want to attach/detach volumes (and so manipulate block device mappings) when 
an instance is not on any compute node (actually when in shelved). Normally 
this happens in a function on the compute manager synchronized on the instance 
uuid. When an instance is in the shelved_offloaded state it is not on a compute 
host, so the operations have to be done at the API (an existing example is when 
the instance deleted in this state - the cleanup is done in the API but is not 
synchronized in this case).

One option I can see is using tack states, using expected_task_state parameter 
in instance.save() to control state transitions. In the API this makes sense as 
the calls will be synchronous so if an operation cannot be done it can be 
reported back to the user in an error return. I'm sure there must be some other 
options.

Any suggestions would be welcome.

Paul

Paul Murray
Nova Technical Lead, HP Cloud
Hewlett Packard Enterprise
+44 117 316 2527



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Migration state machine proposal.

[Murray, Paul (HP Cloud)]

> From: Jay Pipes [mailto:jaypi...@gmail.com]
> On 10/27/2015 01:16 PM, Chris Friesen wrote:
> > On 10/26/2015 06:02 PM, Jay Pipes wrote:
> >
> >> I believe strongly that we should deprecate the existing migrate,
> >> resize, an live-migrate APIs in favor of a single consolidated,
> >> consistent "move"
> >> REST API
> >> that would have the following characteristics:
> >>
> >> * No manual or wait-input states in any FSM graph
> >
> > Sounds good.
> >
> >> * Removal of the term "resize" from the API entirely (the target
> >> resource sizing is an attribute of the move operation, not a
> >> different type of API operation in and of itself)
> >
> > I disagree on this one.
> >
> > As an end-user, if my goal is to take an existing instance and give it
> > bigger disks, my first instinct isn't going to be to look at the "move"
> > operation.  I'm going to look for "scale", or "resize", or something
> > like that.
> >
> > And if an admin wants to migrate an instance away from its current
> > host, why would they want to change its disk size in the process?
> 
> A fair point. However, I think that a generic update VM API, which would
> allow changes to the resources consumed by the VM along with capabiities
> like CPU model or local disk performance (SSD) is a better way to handle this
> than a resize-specific API.


Sorry I am so late to this - but this stuck out for me. 

Resize is an operation that a cloud user would do to his VM. Usually the
cloud user does not know what host the VM is running on so a resize does 
not appear to be a move at all.

Migrate is an operation that a cloud operator does to a VM that is not normally
available to a cloud user. A cloud operator does not change the VM because 
the operator just provides what the user asked for. He only choses where he is 
going to put it.

It seems clear to me that resize and migrate are very definitely different 
things,
even if they are implemented using the same code path internally for 
convenience.
At the very least I believe they need to be kept separate at the API so we can 
apply
different policy to control access to them.



> 
> So, in other words, I'd support this:
> 
> PATCH /servers/
> 
> with some corresponding request payload that would indicate the required
> changes.
> 
> > I do think it makes sense to combine the external APIs for live and
> > cold migration.  Those two are fundamentally similar, logically
> > separated only by whether the instance stays running or not.
> >
> > And I'm perfectly fine with having the internal implementation of all
> > three share a code path, I just don't think it makes sense for the
> > *external* API.
> 
> I think you meant to say you don't think it makes sense to have three
> separate external APIs for what is fundamentally the same operation (move
> a VM), right?
> 
> Best,
> -jay
> 
> >> * Transition to a task-based API for poll-state requests. This means
> >> that in order for a caller to determine the state of a VM the caller
> >> would call something like GET /servers//tasks/ in order
> >> to see the history of state changes or subtask operations for a
> >> particular request to move a VM
> > enstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > Sounds good.
> >
> > Chris
> >

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Migration state machine proposal.

[Murray, Paul (HP Cloud)]

> From: Ed Leafe [mailto:e...@leafe.com]
> On Nov 5, 2015, at 2:43 AM, Tang Chen  wrote:
> 
> > I'm sorry that I cannot understand why resize and migrate are the same
> thing behind.
> 
> Resize is essentially a migration to the same host, rather than a different
> host. The process is still taking an existing VM and using it to create 
> another
> VM that appears to the user as the same (ID, networking, attached volumes,
> metadata, etc.)
> 



Or more specifically, the migrate and resize API actions both call the resize
function in the compute api. As Ed said, they are basically the same behind 
the scenes. (But the API difference is important.)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Custom Nova scheduler based on CPU queue length

[Murray, Paul (HP Cloud)]

Hi Rahul,

> As I understand a weigher for the filter scheduler that weighs filtered host 
> machines based on the host weight option, 'cpu.percent' can be
> configured to prioritize the hosts based on CPU percentage, but there are 
> only limited options when it comes to filtering of machines in the
> first place, that too using the CPU queue length.
>
> Also, I understand that IBM has its platform resource scheduler that can be 
> used to build custom plugins to get user defined metrics, is there
> a similar way in pure Openstack which can be used to get the CPU queue length.
>
> As of now, I am thinking of writing a custom script to be kept in all compute 
> nodes to retrieve the CPU queue length and send it to the Nova
> controller, is this the way to go, or is there a defined approach that I can 
> follow to implement a scheduler that uses CPU queue length to filter
> physical machines.

Nova has metric plugins (monitors) in the resource tracker at the compute 
manager that will report metric data like this to the scheduler.
Any weigher plugins can use that data.

To see how cpu.percentage is set and used see the following plugins:
nova/compute/monitors/cpu_monitor.py
nova/scheduler/weights/metrics.py

You can create new monitor and weigher plugins using these as a model or 
propose an addition to cpu_monitor.py if you think it is generally useful.

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] What is the no_device flag for in block device mapping?

[Murray, Paul (HP Cloud)]

Hi All,

I'm wondering what the "no_device" flag is used for in the block device 
mappings. I had a dig around in the code but couldn't figure out why it is 
there. The name suggests an obvious meaning, but I've learnt not to guess too 
much from names.

Any pointers welcome.

Thanks
Paul

Paul Murray
Nova Technical Lead, HP Cloud
+44 117 316 2527

Hewlett-Packard Limited   |   Registered Office: Cain Road, Bracknell, 
Berkshire, RG12 1HN   |Registered No: 690597 England   |VAT Number: GB 
314 1496 79

This e-mail may contain confidential and/or legally privileged material for the 
sole use of the intended recipient.  If you are not the intended recipient (or 
authorized to receive for the recipient) please contact the sender by reply 
e-mail and delete all copies of this message.  If you are receiving this 
message internally within the Hewlett Packard group of companies, you should 
consider the contents "CONFIDENTIAL".

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Murray, Paul (HP Cloud)]

> Please respond to this post if you have an interest in this and what you 
> would like to see done. 
> Include anything you are already getting on with so we get a clear picture. 

Thank you to those who replied to this thread. I have used the contents to 
start an etherpad page here:

https://etherpad.openstack.org/p/mitaka-live-migration 

I have taken the liberty of listing those that responded to the thread and the 
authors of mentioned patches as interested people.

>From the responses and looking at the specs up for review it looks like there 
>are about five areas that could be addressed in Mitaka and several others that 
>could come later. The first five are:

- migrating instances with a mix of local disks and cinder volumes
- pause instance during migration
- cancel migration
- migrate suspended instances
- improve CI coverage

Not all of these are covered by specs yet and all the existing specs need 
reviews. Please look at the etherpad and see if there is anything you think is 
missing.

Paul



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] live migration in Mitaka

[Murray, Paul (HP Cloud)]

Hi All,

There are various efforts going on around live migration at the moment: fixing 
up CI, bug fixes, additions to cover more corner cases, proposals for new 
operations

Generally live migration could do with a little TLC (see: [1]), so I am going 
to suggest we give some of that care in the next cycle.

Please respond to this post if you have an interest in this and what you would 
like to see done. Include anything you are already getting on with so we get a 
clear picture. If there is enough interest I'll put this together as a proposal 
for a work stream. Something along the lines of "robustify live migration".

Paul

[1]: 
https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/live-migration-at-hp-public-cloud


Paul Murray
Nova Technical Lead, HP Cloud
+44 117 316 2527

Hewlett-Packard Limited   |   Registered Office: Cain Road, Bracknell, 
Berkshire, RG12 1HN   |Registered No: 690597 England   |VAT Number: GB 
314 1496 79

This e-mail may contain confidential and/or legally privileged material for the 
sole use of the intended recipient.  If you are not the intended recipient (or 
authorized to receive for the recipient) please contact the sender by reply 
e-mail and delete all copies of this message.  If you are receiving this 
message internally within the Hewlett Packard group of companies, you should 
consider the contents "CONFIDENTIAL".

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Nova mid cycle details

[Murray, Paul (HP Cloud)]

The Nova Mitaka mid cycle meetup is in Bristol, UK at the Hewlett Packard 
Enterprise office.

The mid cycle wiki page is here:
https://wiki.openstack.org/wiki/Sprints/NovaMitakaSprint

Note that there is a web site for signing up for the event and booking hotel 
rooms at a reduced event rate here:
https://starcite.smarteventscloud.com/hpe/NovaMidCycleMeeting

If you want to book a room at the event rate you do need to register on that 
site.

There is also an Eventbrite event that was created before the above web site 
was available. Do not worry if you have registered using Eventbrite, we will 
recognize those registrations as well. But if you do want to book a room you 
will need to register again on the above site.

Paul

Paul Murray
Nova Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] live migration meeting

[Murray, Paul (HP Cloud)]

Hi All,

The first live migration meeting of the year will be tomorrow - see the meeting 
page: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Agenda on the page, feel free to add.

Happy New Year,
Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Daniel P. Berrange [mailto:berra...@redhat.com]
> Sent: 26 November 2015 17:58
> To: Carlton, Paul (Cloud Services)
> Cc: 少合冯; OpenStack Development Mailing List (not for usage questions);
> John Garbutt; pawel.koniszew...@intel.com; yuntong@intel.com;
> shaohe.f...@intel.com; Murray, Paul (HP Cloud); liyong.q...@intel.com
> Subject: Re: [nova] [RFC] how to enable xbzrle compress for live migration
> 
> On Thu, Nov 26, 2015 at 05:49:50PM +, Paul Carlton wrote:
> > Seems to me the prevailing view is that we should get live migration
> > to figure out the best setting for itself where possible.  There was
> > discussion of being able have a default policy setting that will allow
> > the operator to define balance between speed of migration and impact
> > on the instance.  This could be a global default for the cloud with
> > overriding defaults per aggregate, image, tenant and instance as well
> > as the ability to vary the setting during the migration operation.
> >
> > Seems to me that items like compression should be set in configuration
> > files based on what works best given the cloud operator's environment?
> 
> Merely turning on use of compression is the "easy" bit - there needs to be a
> way to deal with compression cache size allocation, which needs to have
> some smarts in Nova, as there's no usable "one size fits all" value for the
> compression cache size. If we did want to hardcode a compression cache
> size, you'd have to pick set it as a scaling factor against the guest RAM 
> size.
> This is going to be very heavy on memory usage, so there needs careful
> design work to solve the problem of migration compression triggering host
> OOM scenarios, particularly since we can have multiple concurrent
> migrations.
> 


Use cases for live migration generally fall into two types:

1. I need to empty the host (host maintenance/reboot)

2. I generally want to balance load on the cloud

The first case is by far the most common need right now and in that case the 
node gets progressively more empty as VMs are moved off. So the resources 
available for caching etc. grow as the process goes on.

The second case is less likely to be urgent from the operators point of view, 
so doing things more slowly may not be a problem.

So looking at how much resource is available at the start of a migration and 
deciding then what to do on a per VM basis is probably not a bad idea. 
Especially if we can differentiate between the two cases.


> Regards,
> Daniel
> --
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org  -o- http://virt-manager.org :|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Nova mid cycle details

[Murray, Paul (HP Cloud)]

The rates are listed on the hotel information page on that site. All include 
tax and breakfast.

Paul

From: Michael Still [mailto:mi...@stillhq.com]
Sent: 28 November 2015 04:39
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] Nova mid cycle details

Hey,

I filled in the first part of that page, but when it got to hotels I got 
confused. The web site doesn't seem to mention the night rate for the HP price. 
Do you know what that is?

Thanks,
Michael

On Sat, Nov 28, 2015 at 3:06 AM, Murray, Paul (HP Cloud) 
<pmur...@hpe.com<mailto:pmur...@hpe.com>> wrote:
The Nova Mitaka mid cycle meetup is in Bristol, UK at the Hewlett Packard 
Enterprise office.

The mid cycle wiki page is here:
https://wiki.openstack.org/wiki/Sprints/NovaMitakaSprint

Note that there is a web site for signing up for the event and booking hotel 
rooms at a reduced event rate here:
https://starcite.smarteventscloud.com/hpe/NovaMidCycleMeeting

If you want to book a room at the event rate you do need to register on that 
site.

There is also an Eventbrite event that was created before the above web site 
was available. Do not worry if you have registered using Eventbrite, we will 
recognize those registrations as well. But if you do want to book a room you 
will need to register again on the above site.

Paul

Paul Murray
Nova Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527<tel:%2B44%20117%20316%202527>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Rackspace Australia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] [Live Migration] topics for midcycle

[Murray, Paul (HP Cloud)]

Hi All,

Following up on a question raised in the live migration meeting yesterday about 
organizing topics for the mid cycle: please add topic suggestions to 
https://etherpad.openstack.org/p/mitaka-nova-midcycle and we will review them 
in next week's meeting.

Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration meeting tomorrow

[Murray, Paul (HP Cloud)]

The agenda is here: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

With the non-priority spec freeze having passed there is not much on the agenda 
at the moment. If you have anything to add please feel free.

Regards,
Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] theoretical race between live migration and resource audit?

[Murray, Paul (HP Cloud)]

Hi Chris, you are right – and I think there are several synchronization issues 
in the code there.

The migration process should be synchronized as it is in build_run_instance 
etc. At the moment if pre_live_migrate fails due to an RPC timeout (i.e. it 
takes too long for some reason) the rollback can be initiated while it is still 
running. So you can get volume attachments and vif plugging being built and 
torn down at the same time, not good. [I’m not sure if that has a bug report – 
I’ll file one if not].

We need to pay attention to these cases as we refactor the live migration 
process. This has begun with:
https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/async-live-migration-rest-check.html
https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/live_migration_compute_communcation.html

Paul

From: lương hữu tuấn [mailto:tuantulu...@gmail.com]
Sent: 10 June 2016 10:20
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [nova] theoretical race between live migration and 
resource audit?

Hi,

Yes, it is actually a race and we have already faced a negative effect when 
using evacuation. Some information of cpu pinning is lost. Imagine that, in 
some cases, we do some re-scheduling actions (evacuate, live-migration, etc.) 
then immediately do the next actions (delete, resize, etc.) before the 
resource_tracker updates in the next period. In this case, it fails. Actually, 
it has some negative side in writing tests based on the scenario described 
above.

Br,

Tutj

On Fri, Jun 10, 2016 at 10:36 AM, Matthew Booth 
> wrote:
Yes, this is a race.

However, it's my understanding that this is 'ok'. The resource tracker doesn't 
claim to be 100% accurate at all times, right? Otherwise why would it update 
itself in a period task in the first place. It's my understanding that the 
resource tracker is basically a best effort cache, and that scheduling 
decisions can still fail at the host. The resource tracker will fix itself next 
time it runs via its periodic task.

Matt (not a scheduler person)

On Thu, Jun 9, 2016 at 10:41 PM, Chris Friesen 
> wrote:
Hi,

I'm wondering if we might have a race between live migration and the resource 
audit.  I've included a few people on the receiver list that have worked 
directly with this code in the past.

In _update_available_resource() we have code that looks like this:

instances = objects.InstanceList.get_by_host_and_node()
self._update_usage_from_instances()
migrations = objects.MigrationList.get_in_progress_by_host_and_node()
self._update_usage_from_migrations()


In post_live_migration_at_destination() we do this (updating the host and node 
as well as the task state):
instance.host = self.host
instance.task_state = None
instance.node = node_name
instance.save(expected_task_state=task_states.MIGRATING)


And in _post_live_migration() we update the migration status to "completed":
if migrate_data and migrate_data.get('migration'):
migrate_data['migration'].status = 'completed'
migrate_data['migration'].save()


Both of the latter routines are not serialized by the 
COMPUTE_RESOURCE_SEMAPHORE, so they can race relative to the code in 
_update_available_resource().


I'm wondering if we can have a situation like this:

1) migration in progress
2) We start running _update_available_resource() on destination, and we call 
instances = objects.InstanceList.get_by_host_and_node().  This will not return 
the migration, because it is not yet on the destination host.
3) The migration completes and we call post_live_migration_at_destination(), 
which sets the host/node/task_state on the instance.
4) In _update_available_resource() on destination, we call migrations = 
objects.MigrationList.get_in_progress_by_host_and_node().  This will return the 
migration for the instance in question, but when we run 
self._update_usage_from_migrations() the uuid will not be in "instances" and so 
we will use the instance from the newly-queried migration.  We will then ignore 
the instance because it is not in a "migrating" state.

Am I imagining things, or is there a race here?  If so, the negative effects 
would be that the resources of the migrating instance would be "lost", allowing 
a newly-scheduled instance to claim the same resources (PCI devices, pinned 
CPUs, etc.)

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: 

Re: [openstack-dev] [Nova] State machines in Nova

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Monty Taylor [mailto:mord...@inaugust.com]
> Sent: 01 June 2016 13:54
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Nova] State machines in Nova
> 
> On 06/01/2016 03:50 PM, Andrew Laski wrote:
> >
> >
> > On Wed, Jun 1, 2016, at 05:51 AM, Miles Gould wrote:
> >> On 31/05/16 21:03, Timofei Durakov wrote:
> >>> there is blueprint[1] that was approved during Liberty and
> >>> resubmitted to Newton(with spec[2]).
> >>> The idea is to define state machines for operations as
> >>> live-migration, resize, etc. and to deal with them operation states.
> >>
> >> +1 to introducing an explicit state machine - IME they make complex
> >> logic much easier to reason about. However, think carefully about how
> >> you'll make changes to that state machine later. In Ironic, this is
> >> an ongoing problem: every time we change the state machine, we have
> >> to decide whether to lie to older clients (and if so, what lie to
> >> tell them), or whether to present them with the truth (and if so, how
> >> badly they'll break). AIUI this would be a much smaller problem if
> >> we'd considered this possibility carefully at the beginning.
> >
> > This is a great point. I think most people have an implicit assumption
> > that the state machine will be exposed to end users via the API. I
> > would like to avoid that for exactly the reason you've mentioned. Of
> > course we'll want to expose something to users but whatever that is
> > should be loosely coupled with the internal states that actually drive the
> system.
> 
> +1billion
> 

I think this raises an interesting point.

tl;dr: I am starting to think we should not do the migration state machine spec 
being proposed before the tasks. But we should at least make the states we 
assign something other than arbitrary strings (e.g. constants defined in a 
particular place) and we should use the state names consistently.

Transitions can come from two places: 1) the user invokes the API to change the 
state of an instance, this is a good place to check that the instance is in a 
state to do the externally visible transition, 2) the state of the instance 
changes due to an internal event (host crash, deliberate operation...) this 
implies a change in the externally visible state of the instance, but cannot be 
prevented just because the state machine says this shouldn't happen (usually 
this is captured by the error state, but we can do better sometimes).

I think the state machines that are being defined in these changes are actually 
high level phases of the migration process that are currently observed by the 
user. I'm not sure they are particularly useful for coordinating the migration 
process itself and so are maybe not the right place to enforce internal 
transitions.

Live migration is an oddity in nova. Usually an instance is a single entity 
running on a single host (ignoring clustered hypervisors for the moment). There 
is a host manager responsible for that host that has the best view of the 
actual state of the instance or operations being performed on it. Generally the 
host manager is the natural place to coordinate operations on the instance.

In the case of live migration there are actually two VMs running on different 
hosts at a same time. The migration process involves coordinating transitions 
of those two VMs (attaching disks, plugging networks, starting the target VM, 
starting the migration, rebinding ports, stopping the source VM.). The two 
VMs and their own individual states in this process are not represented 
explicitly. We only have an overall process coordinated by a distributed  
sequence of rpcs. There is a current spec moving that coordination to the 
conductor. When that sequence is interrupted or even completely lost (e.g. by a 
conductor failing or being restarted) we get into trouble. I think this is 
where our real problem lies.

We should sort out the internal process. The external view given to the user 
can be a true reflection the current state of the instance. The transitions of 
the instance should be internally coordinated.

Paul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Andrew Laski [mailto:and...@lascii.com]
> Sent: 31 May 2016 22:34
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Nova] State machines in Nova
> 
> 
> 
> On Tue, May 31, 2016, at 04:26 PM, Joshua Harlow wrote:
> > Timofei Durakov wrote:
> > > Hi team,
> > >
> > > there is blueprint[1] that was approved during Liberty and
> > > resubmitted to Newton(with spec[2]).

Do you mean the blueprint was approved - I think it was agreed there was no 
need for a spec then. There is a spec now because there is a minor API impact 
(if state names are changed to be cConsistent across types of migration).



> > > The idea is to define state machines for operations as
> > > live-migration, resize, etc. and to deal with them operation states.
> > > The spec PoC patches are overall good. At the same time I think is
> > > will be good to get agreement on the usage of state-machines in Nova.
> > > There are 2 options:
> > >
> > >   * implement proposed change and use state machines to deal with
> states
> > > only
> >
> > I think this is what could be called the ironic equivalent correct?
> >
> > In ironic @
> >
> https://github.com/openstack/ironic/blob/master/ironic/common/states.p
> > y the state machine here is used to ensure proper states are
> > transitioned over and no invalid/unexpected state transitions happen.
> > The code though itself still runs in a implicit fashion and afaik only
> > interacts with the state machine as a side-effect of actions occurring
> > (instead of the reverse where the state machine itself is 'driving'
> > those actions to happen/to completion).
> 
> Yes. This exists in a limited form already in Nova for instances and
> task_states.
> 
> >
> > >   o procs:
> > >   + could be implemented/merged right now
> > >   + cleans up states for migrations
> > >   o cons:
> > >   + state machine only deal with states, and it will be hard to
> > > build on top of it task API, as bp [1] was designed for
> > > another thing.
> > >
> > >   * use state machines in Task API(which I'm going to work on during
> > > next release):
> >
> > So this would be the second model described above, where the state
> > machine (or set of state machines) itself (together could be formed
> > into a action plan, or action workflow or ...) would be the 'entity'
> > realizing a given action and ensuring that it is performed until
> > completed (or tracking where it was paused and such); is that correct?
> >
> > >   o procs:
> > >   + Task API will orchestrate and deal with long running tasks
> > >   + usage state-machines could help with actions
> > > rollbacks/retries/etc.
> > >   o cons:
> > >   + big amount of work
> > >   + requires time.
> > >
> > > I'd like to discuss these options in this thread.
> >
> > It seems like one could progress from the first model to the second
> > one, although that kind of progression would still be large (because
> > if my understanding is correct the control of who runs what has to be
> > given over to something else in the second model, similar to the
> > control a taskflow engine or mistral engine has over what it runs);
> > said control means that certain programming models may not map so well
> > (from what I have seen).
> 
> I think working through this as a progression from the first model to the
> second one would be the best plan. Start with formalizing the states and
> their allowed transitions and add checking and error handling around that.
> Then work towards handing off control to an engine that could drive the
> operation.
> 

I'm inclined to agree with Andrew. I don't see this as inconsistent with the 
tasks API approach although it may become redundant. The spec is actually 
pretty simple so why make it wait for something that may exist in the future.


> > >
> > > [1] -
> > > https://blueprints.launchpad.net/openstack/?searchtext=migration-sta
> > > te-machine [2] - https://review.openstack.org/#/c/320849/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live Migration meeting

[Murray, Paul (HP Cloud)]

Sorry for late update - this weekend was a holiday in UK

Agenda: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration meeting today

[Murray, Paul (HP Cloud)]

Agenda here: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

We will go over the non-priority feature status today

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Nova midcycle list of attendees

[Murray, Paul (HP Cloud)]

I have created a list of attendees for the Nova midcycle here: 
https://wiki.openstack.org/wiki/Sprints/NovaMitakaSprintAttendees

Obviously I can't put anyone's name on it for privacy reasons. If are attending 
and you would like to let others know when you will be around you might like to 
add yourself. It would also help us with a few logistics too.

Best regards,
Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova][Live Migration] Live migration IRC meeting today

[Murray, Paul (HP Cloud)]

Hi All,

Forgot to hit "save" on agenda yesterday - the agenda is up. Speak to you later.

Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Update on live migration priority

[Murray, Paul (HP Cloud)]

This time with a tag in case anyone is filtering...

From: Murray, Paul (HP Cloud)
Sent: 12 February 2016 16:16
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] Update on live migration priority

The objective for the live migration priority is to improve the stability of 
migrations based on operator experience. The high level approach is to do the 
following:

1.   Improve CI

2.   Improve documentation

3.   Improve manageability of migrations

4.   Fix bugs

In this cycle we targeted a few immediately implementable features that would 
help, specifically giving operators commands to allow them to manage migrations 
(inspect progress, force completion, and cancel) and improve security 
(split-networks and remove ssh-based resize/migration; aka storage pools).

Most of these are on track to be completed in this cycle with the exception of 
storage pools work which is being deferred. Further details follow.

Expand CI coverage - in progress

There is a job in the experimental queue called: 
gate-tempest-dsvm-multinode-live-migrationqueued. This will become the job that 
performs live migration tests; any live migration tests in other jobs will be 
removed. At present the job has been configured to cover different storage 
configurations including cinder, NFS, ceph. Tests are now being added to the 
job. Patches are currently up for live migration of instances with swap and 
instances with ephemeral disks.

Please trigger the experimental queue if your patches touch migrations in some 
way so we can check the stability of the jobs. Once stable and with sufficient 
tests we will promote the job from the experimental queue so that it always 
runs.

See: https://review.openstack.org/#/q/topic:lm_test

Improve API docs - done

Some changes were made to the API guide for moving servers, including better 
descriptions for the server actions migrate, live migrate, shelve, resize and 
evacuate ( 
http://developer.openstack.org/api-guide/compute/server_concepts.html#server-actions
 ) and a section that describes reasons for moving VMs with common use cases 
outlined ( 
http://developer.openstack.org/api-guide/compute/server_concepts.html#moving-servers
 )

Block live migration with attached volumes - done

The selective block device migration API in libvirt 1.2.17 is used to allow 
block migration when volumes are attached. A follow on patch to allow readonly 
drives to be copied in block migration has not been completed. This patch is 
required to allow iso9600 format config drives to be migrated. Without it only 
vfat config drives can be migrated. There is still some thought going into that 
- see: https://review.openstack.org/#/c/234659

Force complete - requires python-novaclient change

Force-complete forces a live migration to complete  by pausing the VM and 
restarting it when it has completed migration. This is intended as a brute 
force way to make a VM complete its migration when it is taking too long. In 
the future auto-converge and post-copy will be looked at. These became 
available in qemu 2.5.

Force complete is done in nova but still requires a change to python-novaclient 
to implement the CLI.

Cancel - in progress

Cancel stops a live migration, leaving it on the source host with the migration 
status left as "cancelled". This is in progress and follows the pattern of 
force-complete. Unfortunately this needs to be bundled up into one patch to 
avoid multiple API bumps.

Patches for review:
https://review.openstack.org/#/q/status:open+topic:bp/abort-live-migration

Progress reporting - in progress (no pun intended)

Progress reporting introduces migrations as a sub-resource of servers and adds 
progress data to the migration record. There was some debate at the mid cycle 
and on the mailing list about how to record this transient data. It is a waste 
to keep writing it to the database, but as it is generated at the compute 
manager but examined at the API it was felt that writing it to the database is 
necessary to fit the existing architecture. The conclusions was that writing to 
the database every 5 seconds would not cause a significant overhead. 
Alternatives could be persued later if necessary. For discussion see this ML 
thread: 
http://lists.openstack.org/pipermail/openstack-dev/2016-February/085662.html 
and the IRC meeting transcript here: 
http://eavesdrop.openstack.org/meetings/nova_live_migration/2016/nova_live_migration.2016-02-09-14.01.log.html

Patches for review:
https://review.openstack.org/#/q/status:open+topic:bp/live-migration-progress-report

Split networking - done

Split networking adds a configuration parameter to specify 
live_migration_inbound_addr as the ip address or host name to be used as the 
target for migration traffic. This allows migration traffic to be isolated on a 
separate network to other management traffic, providing an opportunity to 
islate service levels for the two networks and improve security by moving 
u

[openstack-dev] Update on live migration priority

[Murray, Paul (HP Cloud)]

The objective for the live migration priority is to improve the stability of 
migrations based on operator experience. The high level approach is to do the 
following:

1.   Improve CI

2.   Improve documentation

3.   Improve manageability of migrations

4.   Fix bugs

In this cycle we targeted a few immediately implementable features that would 
help, specifically giving operators commands to allow them to manage migrations 
(inspect progress, force completion, and cancel) and improve security 
(split-networks and remove ssh-based resize/migration; aka storage pools).

Most of these are on track to be completed in this cycle with the exception of 
storage pools work which is being deferred. Further details follow.

Expand CI coverage - in progress

There is a job in the experimental queue called: 
gate-tempest-dsvm-multinode-live-migrationqueued. This will become the job that 
performs live migration tests; any live migration tests in other jobs will be 
removed. At present the job has been configured to cover different storage 
configurations including cinder, NFS, ceph. Tests are now being added to the 
job. Patches are currently up for live migration of instances with swap and 
instances with ephemeral disks.

Please trigger the experimental queue if your patches touch migrations in some 
way so we can check the stability of the jobs. Once stable and with sufficient 
tests we will promote the job from the experimental queue so that it always 
runs.

See: https://review.openstack.org/#/q/topic:lm_test

Improve API docs - done

Some changes were made to the API guide for moving servers, including better 
descriptions for the server actions migrate, live migrate, shelve, resize and 
evacuate ( 
http://developer.openstack.org/api-guide/compute/server_concepts.html#server-actions
 ) and a section that describes reasons for moving VMs with common use cases 
outlined ( 
http://developer.openstack.org/api-guide/compute/server_concepts.html#moving-servers
 )

Block live migration with attached volumes - done

The selective block device migration API in libvirt 1.2.17 is used to allow 
block migration when volumes are attached. A follow on patch to allow readonly 
drives to be copied in block migration has not been completed. This patch is 
required to allow iso9600 format config drives to be migrated. Without it only 
vfat config drives can be migrated. There is still some thought going into that 
- see: https://review.openstack.org/#/c/234659

Force complete - requires python-novaclient change

Force-complete forces a live migration to complete  by pausing the VM and 
restarting it when it has completed migration. This is intended as a brute 
force way to make a VM complete its migration when it is taking too long. In 
the future auto-converge and post-copy will be looked at. These became 
available in qemu 2.5.

Force complete is done in nova but still requires a change to python-novaclient 
to implement the CLI.

Cancel - in progress

Cancel stops a live migration, leaving it on the source host with the migration 
status left as "cancelled". This is in progress and follows the pattern of 
force-complete. Unfortunately this needs to be bundled up into one patch to 
avoid multiple API bumps.

Patches for review:
https://review.openstack.org/#/q/status:open+topic:bp/abort-live-migration

Progress reporting - in progress (no pun intended)

Progress reporting introduces migrations as a sub-resource of servers and adds 
progress data to the migration record. There was some debate at the mid cycle 
and on the mailing list about how to record this transient data. It is a waste 
to keep writing it to the database, but as it is generated at the compute 
manager but examined at the API it was felt that writing it to the database is 
necessary to fit the existing architecture. The conclusions was that writing to 
the database every 5 seconds would not cause a significant overhead. 
Alternatives could be persued later if necessary. For discussion see this ML 
thread: 
http://lists.openstack.org/pipermail/openstack-dev/2016-February/085662.html 
and the IRC meeting transcript here: 
http://eavesdrop.openstack.org/meetings/nova_live_migration/2016/nova_live_migration.2016-02-09-14.01.log.html

Patches for review:
https://review.openstack.org/#/q/status:open+topic:bp/live-migration-progress-report

Split networking - done

Split networking adds a configuration parameter to specify 
live_migration_inbound_addr as the ip address or host name to be used as the 
target for migration traffic. This allows migration traffic to be isolated on a 
separate network to other management traffic, providing an opportunity to 
islate service levels for the two networks and improve security by moving 
unencrypted migration traffic to an isolated network.

Resize/cold migrate using storage pools - deferred

The objective here was to change the libvirt implementation of migrate and 
resize to use libvirt storage pools 

Re: [openstack-dev] [nova] Migration progress

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Daniel P. Berrange [mailto:berra...@redhat.com]
> Sent: 03 February 2016 10:49
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: Feng, Shaohe
> Subject: Re: [openstack-dev] [nova] Migration progress
> 
> On Wed, Feb 03, 2016 at 10:44:36AM +, Daniel P. Berrange wrote:
> > On Wed, Feb 03, 2016 at 10:37:24AM +, Koniszewski, Pawel wrote:
> > > Hello everyone,
> > >
> > > On the yesterday's live migration meeting we had concerns that
> > > interval of writing migration progress to the database is too short.
> > >
> > > Information about migration progress will be stored in the database
> > > and exposed through the API (/servers//migrations/). In
> > > current proposition [1] migration progress will be updated every 2
> > > seconds. It basically means that every 2 seconds a call through RPC
> > > will go from compute to conductor to write migration data to the
> > > database. In case of parallel live migrations each migration will report
> progress by itself.
> > >
> > > Isn't 2 seconds interval too short for updates if the information is
> > > exposed through the API and it requires RPC and DB call to actually
> > > save it in the DB?
> > >
> > > Our default configuration allows only for 1 concurrent live
> > > migration [2], but it might vary between different deployments and
> > > use cases as it is configurable. Someone might want to trigger 10
> > > (or even more) parallel live migrations and each might take even a
> > > day to finish in case of block migration. Also if deployment is big enough
> rabbitmq might be fully-loaded.
> > > I'm not sure whether updating each migration every 2 seconds makes
> > > sense in this case. On the other hand it might be hard to observe
> > > fast enough that migration is stuck if we increase this interval...
> >
> > Do we have any actual data that this is a real problem. I have a
> > pretty hard time believing that a database update of a single field
> > every 2 seconds is going to be what pushes Nova over the edge into a
> > performance collapse, even if there are 20 migrations running in
> > parallel, when you compare it to the amount of DB queries & updates
> > done across other areas of the code for pretty much every singke API call
> and background job.

As a data point: when we were doing live migrations in HP public cloud for 
rolling updates we were maintaining approximately 150 concurrent migrations 
through the process. At 2s intervals that would make approx. 75 updates per 
second. We don't feel that would have been a problem.

We also spoke to Michael Still and he thought it wouldn't be a problem for Rack 
Space (remembering they have cells). Having said that I have no idea of numbers 
I their case and would rather they spoke for themselves. In this thread.



> 
> Also note that progress is rounded to the nearest integer. So even if the
> migration runs all day, there is a maximum of 100 possible changes in value
> for the progress field, so most of the updates should turn in to no-ops at the
> database level.
> 
> Regards,
> Daniel
> --
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org  -o- http://virt-manager.org :|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] mid-cycle arrangements

[Murray, Paul (HP Cloud)]

See updated event detail information for the mid-cycle at: 
https://wiki.openstack.org/wiki/Sprints/NovaMitakaSprint

Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova][Live Migration] no live migration meeting this week

[Murray, Paul (HP Cloud)]

This is just a short reminder that there will be no live migration meeting this 
week due to the mid cycle.

Agenda and other information always available at: 
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] volumes stuck detaching attaching and force detach

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: D'Angelo, Scott
> 
> Matt, changing Nova to store the connector info at volume attach time does
> help. Where the gap will remain is after Nova evacuation or live migration,

This will happen with shelve as well I think. Volumes are not detached in shelve
IIRC.

> when that info will need to be updated in Cinder. We need to change the
> Cinder API to have some mechanism to allow this.
> We'd also like Cinder to store the appropriate info to allow a force-detach 
> for
> the cases where Nova cannot make the call to Cinder.
> Ongoing work for this and related issues is tracked and discussed here:
> https://etherpad.openstack.org/p/cinder-nova-api-changes
> 
> Scott D'Angelo (scottda)
> 
> From: Matt Riedemann [mrie...@linux.vnet.ibm.com]
> Sent: Monday, February 29, 2016 7:48 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [nova][cinder] volumes stuck detaching
> attaching and force detach
> 
> On 2/22/2016 4:08 PM, Walter A. Boring IV wrote:
> > On 02/22/2016 11:24 AM, John Garbutt wrote:
> >> Hi,
> >>
> >> Just came up on IRC, when nova-compute gets killed half way through a
> >> volume attach (i.e. no graceful shutdown), things get stuck in a bad
> >> state, like volumes stuck in the attaching state.
> >>
> >> This looks like a new addition to this conversation:
> >> http://lists.openstack.org/pipermail/openstack-dev/2015-
> December/0826
> >> 83.html
> >>
> >> And brings us back to this discussion:
> >> https://blueprints.launchpad.net/nova/+spec/add-force-detach-to-nova
> >>
> >> What if we move our attention towards automatically recovering from
> >> the above issue? I am wondering if we can look at making our usually
> >> recovery code deal with the above situation:
> >>
> https://github.com/openstack/nova/blob/834b5a9e3a4f8c6ee2e3387845fc24
> >> c79f4bf615/nova/compute/manager.py#L934
> >>
> >>
> >> Did we get the Cinder APIs in place that enable the force-detach? I
> >> think we did and it was this one?
> >> https://blueprints.launchpad.net/python-cinderclient/+spec/nova-force
> >> -detach-needs-cinderclient-api
> >>
> >>
> >> I think diablo_rojo might be able to help dig for any bugs we have
> >> related to this. I just wanted to get this idea out there before I
> >> head out.
> >>
> >> Thanks,
> >> John
> >>
> >>
> __
> ___
> >> _
> >>
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> .
> >>
> > The problem is a little more complicated.
> >
> > In order for cinder backends to be able to do a force detach
> > correctly, the Cinder driver needs to have the correct 'connector'
> > dictionary passed in to terminate_connection.  That connector
> > dictionary is the collection of initiator side information which is gleaned
> here:
> > https://github.com/openstack/os-brick/blob/master/os_brick/initiator/c
> > onnector.py#L99-L144
> >
> >
> > The plan was to save that connector information in the Cinder
> > volume_attachment table.  When a force detach is called, Cinder has
> > the existing connector saved if Nova doesn't have it.  The problem was
> > live migration.  When you migrate to the destination n-cpu host, the
> > connector that Cinder had is now out of date.  There is no API in
> > Cinder today to allow updating an existing attachment.
> >
> > So, the plan at the Mitaka summit was to add this new API, but it
> > required microversions to land, which we still don't have in Cinder's
> > API today.
> >
> >
> > Walt
> >
> >
> __
> 
> >  OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> Regarding storing off the initial connector information from the attach, does
> this [1] help bridge the gap? That adds the connector dict to the
> connection_info dict that is serialized and stored in the nova
> block_device_mappings table, and then in that patch is used to pass it to
> terminate_connection in the case that the host has changed.
> 
> [1] https://review.openstack.org/#/c/266095/
> 
> --
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> 

Re: [openstack-dev] [nova] Nova PTL for Newton

[Murray, Paul (HP Cloud)]

Thanks for a job well done John and for being so available to help us all. 

Paul

> -Original Message-
> From: John Garbutt [mailto:j...@johngarbutt.com]
> Sent: 11 March 2016 19:22
> To: OpenStack Development Mailing List  d...@lists.openstack.org>
> Subject: [openstack-dev] [nova] Nova PTL for Newton
> 
> Hi,
> 
> It has been greatly rewarding serving you all as Nova PTL over the Liberty and
> Mitaka cycles. I thank you all for the support, encouragement and advice I
> have been given over the last year. I really appreciate that. (That's british
> speak for "I love you all", or something like that).
> 
> I don't plan on standing for the Newton cycle. I think its a good time for
> someone to bring fresh energy, ideas, and drive to help keep Nova driving
> forward. I have enjoyed my time as PTL, as such, I would consider standing
> again. We have a good number of folks stepping up to lead different parts of
> the project. I hope we can grow that effort, and I hope to continue to be part
> of that.
> 
> I aim to continue contributing to Nova (I hope to be in Austin, and I hope to
> write some code again soon). I will certainly make time to ensure a smooth
> transition to the next PTL.
> 
> Many thanks,
> johnthetubaguy
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] New resource tracker

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: 04 April 2016 18:47
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [nova] New resource tracker
> 
> On 04/04/2016 01:21 PM, Rayson Ho wrote:
> > I found that even in the latest git tree, the resource_tracker is
> > still marked as "deprecated and will be removed in the 14.0.0" in
> > releasenova/conf/compute.py . With the Mitaka release coming up this
> > week, is it still true that the code will be removed?
> >
> > I googled and found this status update sent to the list (
> > http://lists.openstack.org/pipermail/openstack-dev/2016-February/08637
> > 1.html ), but I was wondering if the new resource tracker is
> > documented or should I just refer to the blueprints.
> 
> The resource tracker has not been deprecated. Only the extensible resource
> tracker code has been deprecated in Mitaka. The extensible resource tracker
> code allowed a deployer to override and extend the set of resources that
> were tracked by the resource tracker. It was determined this functionality
> was leading the scheduler and resource tracker code to a place where it was
> not possible to have consistency in how resources were tracked and thus it
> was deprecated and removed.
> 

See: https://review.openstack.org/#/c/300420/ 

Reviews welcome !

Paul


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Live Migration post feature freeze update

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Sean Dague [mailto:s...@dague.net]
> Sent: 04 March 2016 20:09
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Nova] Live Migration post feature freeze
> update
> 
> On 03/04/2016 02:54 PM, Matt Riedemann wrote:
> >
> >
> > On 3/4/2016 10:34 AM, Murray, Paul (HP Cloud) wrote:
> >> Hi All,
> >>

- snip 8<--

> >
> > The gate-tempest-dsvm-multinode-full job which runs live migration
> > tests on nova patches has been non-voting for awhile now. There are at
> > least two known tracked bugs so we can keep an eye on failure rates.
> >
> > 1. Volume based live migration aborted unexpectedly:
> >
> > http://status.openstack.org/elastic-recheck/index.html#1524898
> >
> > 2. Libvirt live block migration migration stalls
> >
> > http://status.openstack.org/elastic-recheck/index.html#1539271
> >
> > Those are actually the top two failures in the check queue.
> >
> > The job is bouncing between 25% and ~80% failure rates:
> >
> > http://tinyurl.com/gvt5h56
> >
> > At one point that job was relatively stable, it had to have been
> > because it was voting.
> 
> It's never been voting. Or at least not with the number of live migration
> scenarios. The grenade multinode job is the only one that's hit the threshold
> of voting and kept it. It does not run live migration though.
> 
>   -Sean
> 


Thanks Matt, we'll discuss in the meeting tomorrow and see what we can do
about it.

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live Migration post feature freeze update

[Murray, Paul (HP Cloud)]

Hi All,

Now that we have passed the feature freeze I thought it was worth giving a 
quick update
on where we are with the live migration priority.

The following is a list of work items that have been merged in this cycle ( for 
the live migration
sub-team's working page see 
https://etherpad.openstack.org/p/mitaka-live-migration ). There
is also a number of merged and on-going bug fixes that are not listed here.

Progress reporting
Provide progress reporting information for on-going live migrations.

* 
https://blueprints.launchpad.net/nova/+spec/live-migration-progress-report

  *   https://review.openstack.org/#/q/topic:bp/live-migration-progress-report

Force complete
Force an on-going live migration to complete by pausing the virtual machine for 
the
duration of the migration.

* 
https://blueprints.launchpad.net/nova/+spec/pause-vm-during-live-migration

* 
https://review.openstack.org/#/q/topic:bp/pause-vm-during-live-migration

Cancel
Cancel an on-going live migration.

* https://blueprints.launchpad.net/nova/+spec/abort-live-migration

  *   https://review.openstack.org/#/q/topic:bp/abort-live-migration

Block live migration with attached volumes
Enable live migration of VMs with a combination of local and shared storage.

* 
https://blueprints.launchpad.net/nova/+spec/block-live-migrate-with-attached-volumes

* https://review.openstack.org/#/c/227278

Split networking
Send live migration traffic over a specified network.

* 
https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration

* 
https://review.openstack.org/#/q/topic:bp/split-network-plane-for-live-migration

Make live migration api friendly
Remove -disk_over_commit flag and add -block_migration=auto (let nova determine
how to migrate the disks)

* 
https://blueprints.launchpad.net/nova/+spec/making-live-migration-api-friendly

  *   
https://review.openstack.org/#/q/topic:bp/making-live-migration-api-friendly

Use request spec
Add scheduling to live migration and evacuate using original request spec 
(includes all
original scheduling properties)

* 
https://blueprints.launchpad.net/nova/+spec/check-destination-on-migrations

* https://review.openstack.org/#/c/277800/

* https://review.openstack.org/#/c/273104/

Deprecate migration flags
Replace the combination of migration configuration flags with a single tunneled 
flag.

* (no blueprint)

* 
https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:deprecate-migration-flags-config

Objectify live migrate data
Use the migrate object instead of a dictionary in migration code.

* 
https://blueprints.launchpad.net/nova/+spec/objectify-live-migrate-data

* 
https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/objectify-live-migrate-data

Next steps...

Now we have passed the feature freeze we will be turning attention to the 
following
three tasks:

1.   Documenting the new features

2.   Adding expanding the CI coverage

3.   Fixing bugs

The CI job gate-tempest-dsvm-multinode-live-migration was added to the 
experimental
queue earlier In the cycle. We now need to add tests to this job to increase 
coverage. If
you have any suggestions for CI improvements please contribute them on this 
page:

https://etherpad.openstack.org/p/nova-live-migration-CI-ideas

If you can contributed to live migration bug fixing you can look for things to 
do here:

https://bugs.launchpad.net/nova/+bugs?field.tag=live-migration

For priority reviews see the live migration section here:

https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking

The live migration sub-team has an IRC meeting on Tuesdays at 14:00 UTC on
#openstack-meeting-3:

https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Best regards,
Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration meeting today: beware summer time !

[Murray, Paul (HP Cloud)]

The live migration meeting is today at usual time, but beware: a lot of time 
zones have switch to summer time this week.

Agenda: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul

P.S. Sorry for late update - it was a national holiday in UK yesterday
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] live migration meeting Tuesday

[Murray, Paul (HP Cloud)]

The live migration meeting is at the usual time on Tuesday, please feel free to 
add items to the agenda.
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Regards,
Paul

Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [devstack] Adding example "local.conf" files for testing?

[Murray, Paul (HP Cloud)]

Absolutely, yes
+1

Paul

From: ZhiQiang Fan [mailto:aji.zq...@gmail.com]
Sent: 21 April 2016 12:34
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [all] [devstack] Adding example "local.conf" files 
for testing?

+1
more example and documentation are welcomed

On Tue, Apr 19, 2016 at 3:10 AM, John Griffith 
> wrote:


On Thu, Apr 14, 2016 at 1:31 AM, Markus Zoeller 
> wrote:
Sometimes (especially when I try to reproduce bugs) I have the need
to set up a local environment with devstack. Everytime I have to look
at my notes to check which option in the "local.conf" have to be set
for my needs. I'd like to add a folder in devstacks tree which hosts
multiple example local.conf files for different, often used setups.
Something like this:

example-confs
--- newton
--- --- x86-ubuntu-1404
--- --- --- minimum-setup
--- --- --- --- README.rst
--- --- --- --- local.conf
--- --- --- serial-console-setup
--- --- --- --- README.rst
--- --- --- --- local.conf
--- --- --- live-migration-setup
--- --- --- --- README.rst
--- --- --- --- local.conf.controller
--- --- --- --- local.conf.compute1
--- --- --- --- local.conf.compute2
--- --- --- minimal-neutron-setup
--- --- --- --- README.rst
--- --- --- --- local.conf
--- --- s390x-1.1.1-vulcan
--- --- --- minimum-setup
--- --- --- --- README.rst
--- --- --- --- local.conf
--- --- --- live-migration-setup
--- --- --- --- README.rst
--- --- --- --- local.conf.controller
--- --- --- --- local.conf.compute1
--- --- --- --- local.conf.compute2
--- mitaka
--- --- # same structure as master branch. omitted for brevity
--- liberty
--- --- # same structure as master branch. omitted for brevity

Thoughts?

Regards, Markus Zoeller (markus_z)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
​Love the idea personally.  Maybe we could start with a working Neutron multi 
node deployment!!!​


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Live Migration: Austin summit update

[Murray, Paul (HP Cloud)]

Thanks Matt, I meant to cover CI but clearly omitted it. 


> On 30 Apr 2016, at 02:35, Matt Riedemann <mrie...@linux.vnet.ibm.com> wrote:
> 
>> On 4/29/2016 5:32 PM, Murray, Paul (HP Cloud) wrote:
>> The following summarizes status of the main topics relating to live
>> migration after the Newton design summit. Please feel free to correct
>> any inaccuracies or add additional information.
>> 
>> 
>> 
>> Paul
>> 
>> 
>> 
>> -
>> 
>> 
>> 
>> Libvirt storage pools
>> 
>> 
>> 
>> The storage pools work has been selected as one of the project review
>> priorities for Newton.
>> 
>> (see https://etherpad.openstack.org/p/newton-nova-summit-priorities )
>> 
>> 
>> 
>> Continuation of the libvirt storage pools work was discussed in the live
>> migration session. The proposal has grown to include a refactor of the
>> existing libvirt driver instance storage code. Justification for this is
>> based on three factors:
>> 
>> 1.   The code needs to be refactored to use storage pools
>> 
>> 2.   The code is complicated and uses inspection, poor practice
>> 
>> 3.   During the investigation Matt Booth discovered two CVEs in the
>> code – suggesting further work is justified
>> 
>> 
>> 
>> So the proposal is now to follow three stages:
>> 
>> 1.   Refactor the instance storage code
>> 
>> 2.   Adapt to use storage pools for the instance storage
>> 
>> 3.   Use storage pools to drive resize/migration
> 
> We also talked about the need for some additional test coverage for the 
> refactor work:
> 
> 1. A job that uses LVM on the experimental queue.
> 
> 2. ploop should be covered by the Virtuozzo Compute third party CI but we'll 
> need to double-check the test coverage there (is it running the tests that 
> hit the code paths being refactored). Note that they have their own blueprint 
> for implementing resize for ploop:
> 
> https://blueprints.launchpad.net/nova/+spec/virtuozzo-instance-resize-support
> 
> 3. Ceph testing - we already have a single-node job for Ceph that will test 
> the resize paths. We should also be testing Ceph-backed live migration in the 
> special live-migration job that Timofey has been working on.
> 
> 4. NFS testing - this also falls into the special live migration CI job that 
> will test live migration in different storage configurations within a single 
> run.
> 
>> 
>> 
>> 
>> Matt has code already starting the refactor and will continue with help
>> from Paul Carlton + Paul Murray. We will look for additional
>> contributors to help as we plan out the patches.
>> 
>> 
>> 
>> https://review.openstack.org/#/c/302117 : Persist libvirt instance
>> storage metadata
>> 
>> https://review.openstack.org/#/c/310505 : Use libvirt storage pools
>> 
>> https://review.openstack.org/#/c/310538 : Migrate libvirt volumes
>> 
>> 
>> 
>> Post copy
>> 
>> 
>> 
>> The spec to add post copy migration support in the libvirt driver was
>> discussed in the live migration session. Post copy guarantees completion
>> of a migration in linear time without needing to pause the VM. This can
>> be used as an alternative to pausing in live-migration-force-complete.
>> Pause or complete could also be invoked automatically under some
>> circumstances. The issue slowing these specs is how to decide which
>> method to use given they provide a different user experience but we
>> don’t want to expose virt specific features in the API. Two additional
>> specs listed below suggest possible generic ways to address the issue.
>> 
>> 
>> 
>> There was no conclusions reached in the session so the debate will
>> continue on the specs. The first below is the main spec for the feature.
>> 
>> 
>> 
>> https://review.openstack.org/#/c/301509 : Adds post-copy live migration
>> support to Nova
>> 
>> https://review.openstack.org/#/c/305425 : Define instance availability
>> profiles
>> 
>> https://review.openstack.org/#/c/306561 : Automatic Live Migration
>> Completion
>> 
>> 
>> 
>> Live Migration orchestrated via conductor
>> 
>> 
>> 
>> The proposal to move orchestration of live migration to conductor was
>> discussed in the working session on Friday, presented by Andrew Laski on
>> behalf of Timofey Durakov. This one threw up a lot of debate both for
>> and a

Re: [openstack-dev] [nova] Thoughts on deprecating the legacy bdm v1 API support

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> Sent: 17 May 2016 15:48
> To: OpenStack Development Mailing List (not for usage questions)
> 
> Subject: [openstack-dev] [nova] Thoughts on deprecating the legacy bdm v1
> API support
> 
> In the live migration meeting today mdbooth and I were chatting about how
> hard it is to follow the various BDM code through nova, because you have
> the three block_device modules:
> 
> * nova.block_device - dict that does some translation magic
> * nova.objects.block_device - contains the BDM(List) objects for RPC and DB
> access
> * nova.virt.block_device - dict that wraps a BDM object, used for attaching
> volumes to instances, updates the BDM.connection_info field in the DB via
> the wrapper on the BDM object. This module also has translation logic in it.
> 
> The BDM v1 extension translates that type of request to the BDM v2 model
> before it gets to server create, and then passed down to the
> nova.compute.api. But there is still a lot of legacy bdm v1 translation logic
> spread through the code.
> 
> So I'd like to propose that we deprecate the v1 BDM API in the same vein
> that we're deprecating other untested things, like agent-builds, cloudpipe,
> certificates, and the proxy APIs. We can't remove the code, but we can signal
> to users to not use the API and eventually when we raise the minimum
> required microversion >= the deprecation, we can drop that code. Since
> that's a long ways off, the earlier we start a deprecation clock on this the
> better - if we're going to do it.
> 
> Does anyone have any issues with this?
> 


No issues, let's do it.

Paul Murray

> --
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] change console proxy URL for cells v2 ?

[Murray, Paul (HP Cloud)]

I am implementing the following spec and have encountered a problem related
to the form of the console proxy URL provided to uses.

https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/convert-consoles-to-objects.html
 

The spec changes the representation of the console auth token connection 
information
to be an object stored in the database, instead of the existing dict stored in 
memcache 
in the console auth server.

At the same time we are rolling out the changes for cells v2. After talking to 
alaski
we have decided that the tokens should go in the child cell database because the
tokens relate to instances.

This introduces a small problem. The console URL provided to users only includes
a token, so this is not enough to map to the cell and database that contains the
record related to the token.

The URL for the console proxy is determined by the compute nodes according to a
configuration parameter. So one solution to this problem is to have a different 
proxy
exposed for each cell and configure the compute nodes with the URL for the
appropriate proxy for its cell.

Another solution, which can actually be implemented after the first, could be 
to add
some information to the URL resource path that can be used by the proxy to map 
to
the cell and database containing the token record.

So the question is: what would be an appropriate addition to the URL resource 
path
that can be used to map to the cell? 
- it could be a UUID for the instance (my favourite)
- it could be something directly related to the cell
- it could be something else altogether

Does anyone have a better suggestion or a preference?

Paul


Paul Murray
Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration meeting today

[Murray, Paul (HP Cloud)]

Agenda: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Please add items if you want to guarantee time to talk about them.

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live Migration: Austin summit update

[Murray, Paul (HP Cloud)]

The following summarizes status of the main topics relating to live migration 
after the Newton design summit. Please feel free to correct any inaccuracies or 
add additional information.

Paul

-

Libvirt storage pools

The storage pools work has been selected as one of the project review 
priorities for Newton.
(see https://etherpad.openstack.org/p/newton-nova-summit-priorities )

Continuation of the libvirt storage pools work was discussed in the live 
migration session. The proposal has grown to include a refactor of the existing 
libvirt driver instance storage code. Justification for this is based on three 
factors:

1.   The code needs to be refactored to use storage pools

2.   The code is complicated and uses inspection, poor practice

3.   During the investigation Matt Booth discovered two CVEs in the code - 
suggesting further work is justified

So the proposal is now to follow three stages:

1.   Refactor the instance storage code

2.   Adapt to use storage pools for the instance storage

3.   Use storage pools to drive resize/migration

Matt has code already starting the refactor and will continue with help from 
Paul Carlton + Paul Murray. We will look for additional contributors to help as 
we plan out the patches.

https://review.openstack.org/#/c/302117 : Persist libvirt instance storage 
metadata
https://review.openstack.org/#/c/310505 : Use libvirt storage pools
https://review.openstack.org/#/c/310538 : Migrate libvirt volumes

Post copy

The spec to add post copy migration support in the libvirt driver was discussed 
in the live migration session. Post copy guarantees completion of a migration 
in linear time without needing to pause the VM. This can be used as an 
alternative to pausing in live-migration-force-complete. Pause or complete 
could also be invoked automatically under some circumstances. The issue slowing 
these specs is how to decide which method to use given they provide a different 
user experience but we don't want to expose virt specific features in the API. 
Two additional specs listed below suggest possible generic ways to address the 
issue.

There was no conclusions reached in the session so the debate will continue on 
the specs. The first below is the main spec for the feature.

https://review.openstack.org/#/c/301509 : Adds post-copy live migration support 
to Nova
https://review.openstack.org/#/c/305425 : Define instance availability profiles
https://review.openstack.org/#/c/306561 : Automatic Live Migration Completion

Live Migration orchestrated via conductor

The proposal to move orchestration of live migration to conductor was discussed 
in the working session on Friday, presented by Andrew Laski on behalf of 
Timofey Durakov. This one threw up a lot of debate both for and against the 
general idea, but not supporting the patches that have been submitted along 
with the spec so far. The general feeling was that we need to attack this, but 
need to take some simple first cleanup steps first to get a better idea of the 
problem. Dan Smith proposed moving the stateless pre-migration steps to a 
sequence of calls from conductor (as opposed to the going back and forth 
between computes) as the first step.

https://review.openstack.org/#/c/292271 : Remove compute-compute communication 
in live-migration

Cold and Live Migration Scheduling

When this patch merges all migrations will use the request spec for scheduling: 
https://review.openstack.org/#/c/284974
Work is still ongoing for check destinations (allowing the scheduler to check a 
destination chosen by the admin). When that is complete migrations will have 
three ways to be placed:

1.   Destination chosen by scheduler

2.   Destination chosen by admin but checked by scheduler

3.   Destination forced by admin

https://review.openstack.org/#/c/296408 Re-Proposes to check destination on 
migrations

PCI + NUMA claims

Moshe and Jay are making great progress refactoring Nicola's patches to fix PCI 
and NUMA handling in migrations. The patch series should be completed soon.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration meeting today

[Murray, Paul (HP Cloud)]

First live migration meeting to kick off after the summit will be today, see: 
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Should be instance_dir in all nova compute node same ?

[Murray, Paul (HP Cloud)]

The instance path is assumed to be the same everywhere. When using libvirt to 
do cold migration/resize the source host creates the instance directory on the 
target host using ssh and then copies the files to it using scp. So the target 
host doesn’t even get to determine where its own files are created. (note: this 
is one of the security flaws we intend to remove in the libvirt storage pools 
work).


From: Michael Still [mailto:mi...@stillhq.com]
Sent: 04 May 2016 06:16
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [nova] Should be instance_dir in all nova compute 
node same ?

I can't think of a reason. In fact its a bit warty because we've changed the 
way we name the instance directories at least once. Its just how this code was 
written back in the day.

Cleaning this up would be a fair bit of work though. Is it really worth the 
effort just so people can have different directory names for some undefined 
reason?

Michael



On Wed, May 4, 2016 at 9:03 AM, Ghe Rivero 
> wrote:
Is there any specific reason why this is require that way? or just a "feature"?

Ghe Rivero

On 03/05/16 11:42, Matthew Booth wrote:
On Fri, Apr 29, 2016 at 2:47 AM, Eli Qiao 
> wrote:
hi team,

Is there any require that all compute node's instance_dir should be same?

Yes. This is assumed in many places, certainly in cold migration/resize.

Matt
--
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)



__

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Rackspace Australia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][osc] Use of openstack client for admin commands

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Edward Leafe [mailto:e...@leafe.com]
> Sent: 05 May 2016 04:32
> To: OpenStack Development Mailing List (not for usage questions)
> 
> Subject: Re: [openstack-dev] [nova][osc] Use of openstack client for admin
> commands
> 
> On May 4, 2016, at 3:53 PM, Jay Pipes  wrote:
> 
> > My position is that if it's an HTTP API (as opposed to something like a
> sqlalchemy-migrate sync command) then it belongs in a client that speaks the
> OpenStack HTTP APIs. That is OSC as far as I can tell. I don't see a 
> difference
> between "admin" commands and "standard" commands.
> 
> Exactly. If it's an admin command and you're not an admin, you get an error.
> No need for a separate client.
> 

And more to the point, as far as I can tell it's only a statement in policy 
that determines which role can use the command. Any operator can change the 
policy and suddenly any division between clients becomes pure nonsense.


> 
> -- Ed Leafe
> 
> 
> 
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration IRC meeting on 26th July

[Murray, Paul (HP Cloud)]

The next live migration meeting is on 26th July.

For agenda see: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration 
(still being updated)

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Quick poll for live migration meeting this week

[Murray, Paul (HP Cloud)]

Just to confirm, next meeting will be after the mid cycle, I will update the 
agenda nearer the time: 
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul

From: Murray, Paul (HP Cloud)
Sent: 11 July 2016 13:32
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Nova] Quick poll for live migration meeting this week

Hi All,

I think we can skip the live migration meeting this week unless anyone has 
something specific to discuss. Next week is the mid cycle, so I suggest the 
next meeting should be Tuesday 26th July.

Please let me know if you agree or disagree. Unless I hear otherwise I will 
assume its ok to kick off again after the mid cycle.

Regards,
Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova][SR-IOV][pci-passthrough] Reporting pci devices in hypervisor-show

[Murray, Paul (HP Cloud)]

Hi All,

At the moment I am not aware of a nova api call that provides information about 
the pci devices on a host. The most obvious place to put this would be in 
hypervisor-show. I wonder if anyone has made an attempt at this already or if 
there are any reasons for not adding pci information there?

Assuming pci device information was put in hypervisor-show I would be 
interested in how people think it would be presented. There are different types 
of pci device and things like virtual functions and parent device 
relationships. The information should include the allocation status.

If hypervisor-show is not the place for this I would be interested in 
suggestions on where it should go.

Paul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Quick poll for live migration meeting this week

[Murray, Paul (HP Cloud)]

Hi All,

I think we can skip the live migration meeting this week unless anyone has 
something specific to discuss. Next week is the mid cycle, so I suggest the 
next meeting should be Tuesday 26th July.

Please let me know if you agree or disagree. Unless I hear otherwise I will 
assume its ok to kick off again after the mid cycle.

Regards,
Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Neutron][Live-Migration] Cross l2 agent migration and solving Nova-Neutron live migration bugs

[Murray, Paul (HP Cloud)]

> -Original Message-
> From: Carl Baldwin [mailto:c...@ecbaldwin.net]
> Sent: 29 June 2016 22:20
> To: OpenStack Development Mailing List (not for usage questions)
> 
> Subject: Re: [openstack-dev] [Nova][Neutron][Live-Migration] Cross l2 agent
> migration and solving Nova-Neutron live migration bugs
> 
> On Tue, Jun 28, 2016 at 9:42 AM, Andreas Scheuring
>  wrote:
> > I'm currently working on solving Nova-Neutron issues during Live
> > Migration. This mail is intended to raise awareness cross project and
> > get things kicked off.
> 
> Thanks for sending this.
> 
> > The issues
> > ==
> >
> > #1 When portbinding fails, instance is migrated but stuck in error
> > state
> > #2 Macvtap agent live Migration when source and target use different
> > physical_interface_mapping [3]. Either the migration fails (good case)
> > or it would place the instance on a wrong network (worst case)
> > #3 (more a feature):  Migration cross l2 agent not possible (e.g.
> > migrate from lb host to ovs host, or from ovs-hybrid to new
> > ovsfirewall
> > host)
> 
> Since all of these issues are experienced when using live migration, a Nova
> feature, I was interested in finding out how Nova prioritizes getting a fix 
> and
> then trying to align Neutron's priority to that priority.  It would seem 
> artificial
> to me to drive priority from Neutron.  That's why I mentioned it.  Since Nova
> is hitting a freeze for non-priority work tomorrow, I don't think anything can
> be done for Newton.  However, there is a lot of time to tee up this
> conversation for Ocata if we get on it.
> 

I read the comments from the neutron meeting here: 
http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-06-30-22.00.log.html#l-320
 and thought a little commentary on our priorities over the last couple of 
cycles might avoid any misconception.

Live migration was a priority in Mitaka because it simply was not up to scratch 
for production use. The main objective was to make it work and make it 
manageable. That translated to:

1. Expand CI coverage
2. Manage migrations: monitor progress, cancel migrations, force spinning 
migrations to complete
3. Extend use cases: allow mix of volumes, shared storage and local disks to be 
migrated
4. Some other things: simplify config and APIs, scheduling support, separate 
migration traffic from management network

These were mostly covered including some supporting work on qemu and libvirt as 
well.

We next wanted to do some security work (refactoring image backend and removing 
ssh-based copy - aka storage pools) but that could not be done in Mitaka and 
was deferred. The priority for Newton was specifically this security work and 
continuing efforts on CI which is now making progress (ref: the sterling work 
by Daniel Berrange in the last couple of days: 
http://lists.openstack.org/pipermail/openstack-dev/2016-June/098540.html )

 
> > The proposal
> > 
> > All those problems could be solved with the same approach . The
> > proposal is, to bind a port to the source AND to the target port
> > during migration.
> >

Good - it is frankly ridiculous that we irreversibly commit a migration to the 
destination host before we know the networks can be built out. This is 
basically what we do with cinder - volumes are attached at both source and 
destination during the migration.


> > * Neutron would need to allow multiple bindings for a compute port and
> > externalize that via API.
> >   - Neutron Spec [1]
> >   - Bug [4]  is a prereq to the spec.
> >
> > * Nova would need to use those new APIs to check in
> > pre_live_migration, if the binding for target host is valid and to
> > modify the instance definition (e.g. domain.xml) during migration.
> >   - Nova Spec [2]
> >
> > This would solve the issues in the following way:
> > #1 would abort the migration before it started, so instance is still
> > usable
> > #2 Migration is possible with all configurations
> > #3 would allow such a migration
> >
> > Coordination
> > 
> > Some coordination between Nova & Neutron is required. Along todays
> > Nova Live Migration Meeting [5] this will happen on the Nova midcycle.
> > I put an item on the agenda [6].
> 
> I'll be there.

Yes, Good. 


> 
> > Would be great that anybody that is interested in this bugfix/feature
> > could comment on the specs [1] or [2] to get as much feedback as
> > possible before the nova midcycle in July!
> >
> > Thank you!
> 

I don't think this is actually a large amount of work on the Nova side. We need 
to get the API right - will leave comments.


> >
> > [1] Neutron spec: https://review.openstack.org/#/c/309416
> > [2] Nova spec: https://review.openstack.org/301090
> > [3] macvtap bug: https://bugs.launchpad.net/neutron/+bug/1550400
> > [4] https://bugs.launchpad.net/neutron/+bug/1367391
> > [5]
> >
> 

[openstack-dev] [Nova] live migration meeting today

[Murray, Paul (HP Cloud)]

See agenda and add if you have something to bring up: 
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration IRC meeting

[Murray, Paul (HP Cloud)]

There will be a live migration sub team meeting today, see:

https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Iive migration IRC meeting

[Murray, Paul (HP Cloud)]

There will be a meeting today – just amended the agenda:
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration IRC meeting

[Murray, Paul (HP Cloud)]

There will be a live migration meeting today – agenda and time here:
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Murray, Paul (HP Cloud)]

From: Matthew Booth
Reply-To: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>"
Date: Friday, 30 September 2016 at 17:03
To: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>"
Subject: Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML 
canonical

On Fri, Sep 30, 2016 at 4:38 PM, Murray, Paul (HP Cloud) 
<pmur...@hpe.com<mailto:pmur...@hpe.com>> wrote:





On 27/09/2016, 18:12, "Daniel P. Berrange" 
<berra...@redhat.com<mailto:berra...@redhat.com>> wrote:

>On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
>> On 09/27/2016 10:17 AM, Matthew Booth wrote:
>>
>> > I think we should be able to create a domain, but once created we should 
>> > never
>> > redefine a domain. We can do adding and removing devices dynamically using
>> > libvirt's apis, secure in the knowledge that libvirt will persist this for 
>> > us.
>> > When we upgrade the host, libvirt can ensure we don't break guests which 
>> > are on
>> > it. Evacuate should be pretty much the only reason to start again.
>>
>> Sounds interesting.  How would you handle live migration?
>>
>> Currently we regenerate the XML file on the destination from the nova DB.  I
>> guess in your proposal we'd need some way of copying the XML file from the
>> source to the dest, and then modifying the appropriate segments to adjust
>> things like CPU/NUMA pinning?
>
>Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
>new persistent XML on the target host automatically.

We have a problem migrating rescued instances that has a fix in progress based
on regenerating the xml on unrescue, see:

 https://blueprints.launchpad.net/nova/+spec/live-migrate-rescued-instances

That might be another case for generating the xml.

I thought it was a counter-argument (unless I've misunderstood). If you migrate 
the instance as-is without modification, you don't need to worry about whether 
it's currently a rescue instance. This problem goes away.

The major complication I can think of is things which genuinely must change 
during a migration, for example cpu pinning.

Rescue currently saves the libvirt xml in a separate file and replaces it with 
new xml to define a vm with a rescue boot disk; to unrescue it replaces the 
libvirt xml used for the rescue with the saved file to go back to the original 
state. On migration that saved xml would be lost because its an arbitrary file 
that is not handled in the migration. The spec above relies on the fact that we 
do not need to save it or copy it because we can recreate it from nova. So yes, 
the migration just works for the rescued vm, but when it is unrescued the 
original libvirt xml would be regenerated.

Paul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Murray, Paul (HP Cloud)]

On 27/09/2016, 18:12, "Daniel P. Berrange"  wrote:

>On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
>> On 09/27/2016 10:17 AM, Matthew Booth wrote:
>> 
>> > I think we should be able to create a domain, but once created we should 
>> > never
>> > redefine a domain. We can do adding and removing devices dynamically using
>> > libvirt's apis, secure in the knowledge that libvirt will persist this for 
>> > us.
>> > When we upgrade the host, libvirt can ensure we don't break guests which 
>> > are on
>> > it. Evacuate should be pretty much the only reason to start again.
>> 
>> Sounds interesting.  How would you handle live migration?
>> 
>> Currently we regenerate the XML file on the destination from the nova DB.  I
>> guess in your proposal we'd need some way of copying the XML file from the
>> source to the dest, and then modifying the appropriate segments to adjust
>> things like CPU/NUMA pinning?
>
>Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
>new persistent XML on the target host automatically.

We have a problem migrating rescued instances that has a fix in progress based
on regenerating the xml on unrescue, see:

 https://blueprints.launchpad.net/nova/+spec/live-migrate-rescued-instances 

That might be another case for generating the xml.

Paul


>
>Regards,
>Daniel
>-- 
>|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
>|: http://libvirt.org  -o- http://virt-manager.org :|
>|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Live migration sub-team lead

[Murray, Paul (HP Cloud)]

Hi All,

I have had the pleasure of chairing the live migration sub-team for the last 
year. Recently, as some of you will have noticed, my time has been stretched to 
the extent that I have started to neglect the task. Timofei Durakov has stood 
in for me on several occasions recently and has now agreed to take over full 
time. I had planned to bring this up in the meeting tomorrow, but yet again I 
am unable to attend.

So thanks to all in the sub-team for being so self-organising and making the 
meetings so easy. Please turn to Timofei as your first point of contact for all 
things live migration from now on.

Best regards,
Paul

P.S. I will be in the meetings when I can.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Detect changes in object model

[Murray, Paul (HP Cloud Services)]

Sounds good to me. The list base objects don't have methods to make changes to 
the list - so it would be a case of iterating looking at each object in the 
list. That would be ok. 

Do we need the contents of the lists to be modified without assigning a new 
list? - that would need a little more work to allow the changes and to track 
them there too.

Paul.

-Original Message-
From: Dan Smith [mailto:d...@danplanet.com] 
Sent: 10 January 2014 14:42
To: Wang, Shane; OpenStack Development Mailing List (not for usage questions)
Cc: Murray, Paul (HP Cloud Services); Lee, Alexis; Tan, Lin
Subject: Re: [Nova] Detect changes in object model

 If an object A contains another object or object list (called 
 sub-object), any change happened in the sub-object can't be detected 
 by obj_what_changed() in object A.

Well, like the Instance object does, you can override obj_what_changed() to 
expose that fact to the caller. However, I think it might be good to expand the 
base class to check, for any NovaObject fields, for the
obj_what_changed() of the child.

How does that sound?

--Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Detect changes in object model

[Murray, Paul (HP Cloud Services)]

Hi Dan,

I was actually thinking of changes to the list itself rather than the objects 
in the list. To try and be clear, I actually mean the following:

ObjectListBase has a field called objects that is typed 
fields.ListOfObjectsField('NovaObject'). I can see methods for count and index, 
and I guess you are talking about adding a method for are any of your contents 
changed here. I don't see other list operations (like append, insert, remove, 
pop) that modify the list. If these were included they would have to mark the 
list as changed so it is picked up when looking for changes. 

Do you see these belonging here or would you expect those to go in a sub-class 
if they were wanted?

Paul.

-Original Message-
From: Dan Smith [mailto:d...@danplanet.com] 
Sent: 10 January 2014 16:22
To: Murray, Paul (HP Cloud Services); Wang, Shane; OpenStack Development 
Mailing List (not for usage questions)
Cc: Lee, Alexis; Tan, Lin
Subject: Re: [Nova] Detect changes in object model

 Sounds good to me. The list base objects don't have methods to make changes 
 to the list - so it would be a case of iterating looking at each object in 
 the list. That would be ok. 

Hmm? You mean for NovaObjects that are lists? I hesitate to expose lists as 
changed when one of the objects inside has changed because I think that sends 
the wrong message. However, I think it makes sense to have a different method 
on lists for are any of your contents changed?

I'll cook up a patch to implement what I'm talking about so you can take a look.

--Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Detect changes in object model

[Murray, Paul (HP Cloud Services)]

Yes, I agree. 

Actually, I am trying to infer what the programming model for this is as we 
go along. 

Personally I would have been happy with only marking the fields when they are 
set. Then, if a you want to change a list somehow you would get it and then set 
it again, e.g.: 
  mylist = object.alist 
  do something to mylist
  object.alist = mylist
  object.save()

Having said that, it can be convenient to use the data structures in place. In 
which case we need all these means to track the changes and they should go in 
the base classes so they are used consistently.

So in short, I am happy with your dirty children :)

Paul.

-Original Message-
From: Dan Smith [mailto:d...@danplanet.com] 
Sent: 13 January 2014 15:26
To: Murray, Paul (HP Cloud Services); Wang, Shane; OpenStack Development 
Mailing List (not for usage questions)
Cc: Lee, Alexis; Tan, Lin
Subject: Re: [Nova] Detect changes in object model

 ObjectListBase has a field called objects that is typed 
 fields.ListOfObjectsField('NovaObject'). I can see methods for count 
 and index, and I guess you are talking about adding a method for are 
 any of your contents changed here. I don't see other list operations 
 (like append, insert, remove, pop) that modify the list. If these were 
 included they would have to mark the list as changed so it is picked 
 up when looking for changes.
 
 Do you see these belonging here or would you expect those to go in a 
 sub-class if they were wanted?

Well, I've been trying to avoid implying the notion that a list of things 
represents the content of the database. Meaning, I don't think it makes sense 
for someone to get a list of Foo objects, add another Foo to the list and then 
call save() on the list. I think that ends up with the assumption that the list 
matches the contents of the database, and if I add or remove things from the 
list, I can save() the contents to the database atomically. That definitely 
isn't something we can or would want to support.

That said, if we make the parent object consider the child to be dirty if any 
of its contents are dirty or the list itself is dirty (i.e. the list of objects 
has changed) that should give us the desired behavior for change tracking, 
right?

--Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] how is resource tracking supposed to work for live migration and evacuation?

[Murray, Paul (HP Cloud Services)]

To be clear - the changes that Yunhong describes below are not part of the 
extensible-resource-tracking blueprint. Extensible-resource-tracking has the 
more modest aim to provide plugins to track additional resource data.

Paul.

-Original Message-
From: Jiang, Yunhong [mailto:yunhong.ji...@intel.com] 
Sent: 17 January 2014 05:54
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova] how is resource tracking supposed to work 
for live migration and evacuation?

There are some related discussion on this before. 

There is a BP at 
https://blueprints.launchpad.net/nova/+spec/extensible-resource-tracking which 
try to support more resources.

And I have a documentation at  
https://docs.google.com/document/d/1gI_GE0-H637lTRIyn2UPfQVebfk5QjDi6ohObt6MIc0 
. My idea is to keep the claim as an object which can be invoked remotely, and 
the claim result is kept in DB as the instance's usage. I'm working on it now.

Thanks
--jyh

 -Original Message-
 From: Vishvananda Ishaya [mailto:vishvana...@gmail.com]
 Sent: Thursday, January 16, 2014 2:27 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [nova] how is resource tracking supposed 
 to work for live migration and evacuation?
 
 
 On Jan 16, 2014, at 1:12 PM, Chris Friesen 
 chris.frie...@windriver.com
 wrote:
 
  Hi,
 
  I'm trying to figure out how resource tracking is intended to work 
  for live
 migration and evacuation.
 
  For a while I thought that maybe we were relying on the call to
 ComputeManager._instance_update() in
 ComputeManager.post_live_migration_at_destination().  However, in
  ResourceTracker.update_usage() we see that on a live migration the
 instance that has just migrated over isn't listed in 
 self.tracked_instances and so we don't actually update its usage.
 
  As far as I can see, the current code will just wait for the audit 
  to run at
 some unknown time in the future and call update_available_resource(), 
 which will add the newly-migrated instance to self.tracked_instances 
 and update the resource usage.
 
  From my poking around so far the same thing holds true for 
  evacuation
 as well.
 
  In either case, just waiting for the audit seems somewhat haphazard.
 
  Would it make sense to do something like
 ResourceTracker.instance_claim() during the migration/evacuate and 
 properly track the resources rather than wait for the audit?
 
 Yes that makes sense to me. Live migration was around before we had a 
 resource tracker so it probably was just never updated.
 
 Vish
 
 
  Chris
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] bp proposal: discovery of peer instances through metadata service

[Murray, Paul (HP Cloud Services)]

Hi Justin,

It's nice to see someone bringing this kind of thing up. Seeding discovery is a 
handy primitive to have.

Multicast is not generally used over the internet, so the comment about 
removing multicast is not really justified, and any of the approaches that work 
there could be used. Alternatively your instances could use the nova or neutron 
APIs to obtain any information you want - if they are network connected - but 
certainly whatever is starting them has access, so something can at least 
provide the information.

I agree that the metadata service is a sensible alternative. Do you imagine 
your instances all having access to the same metadata service? Is there 
something more generic and not tied to the architecture of a single openstack 
deployment?

Although this is a simple example, it is also the first of quite a lot of 
useful primitives that are commonly provided by configuration services. As it 
is possible to do what you want by other means (including using an 
implementation that has multicast within subnets - I'm sure neutron does 
actually have this), it seems that this makes less of a special case and rather 
a requirement for a more general notification service?

Having said that I do like this kind of stuff :)

Paul.


From: Justin Santa Barbara [mailto:jus...@fathomdb.com]
Sent: 24 January 2014 15:43
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] bp proposal: discovery of peer instances 
through metadata service

Good points - thank you.  For arbitrary operations, I agree that it would be 
better to expose a token in the metadata service, rather than allowing the 
metadata service to expose unbounded amounts of API functionality.  We should 
therefore also have a per-instance token in the metadata, though I don't see 
Keystone getting the prerequisite IAM-level functionality for two+ releases (?).

However, I think I can justify peer discovery as the 'one exception'.  Here's 
why: discovery of peers is widely used for self-configuring clustered services, 
including those built in pre-cloud days.  Multicast/broadcast used to be the 
solution, but cloud broke that.  The cloud is supposed to be about distributed 
systems, yet we broke the primary way distributed systems do peer discovery. 
Today's workarounds are pretty terrible, e.g. uploading to an S3 bucket, or 
sharing EC2 credentials with the instance (tolerable now with IAM, but painful 
to configure).  We're not talking about allowing instances to program the 
architecture (e.g. attach volumes etc), but rather just to do the equivalent of 
a multicast for discovery.  In other words, we're restoring some functionality 
we took away (discovery via multicast) rather than adding 
programmable-infrastructure cloud functionality.

We expect the instances to start a gossip protocol to determine who is actually 
up/down, who else is in the cluster, etc.  As such, we don't need accurate 
information - we only have to help a node find one living peer.  
(Multicast/broadcast was not entirely reliable either!)  Further, instance #2 
will contact instance #1, so it doesn't matter if instance #1 doesn't have 
instance #2 in the list, as long as instance #2 sees instance #1.  I'm relying 
on the idea that instance launching takes time  0, so other instances will be 
in the starting state when the metadata request comes in, even if we launch 
instances simultaneously.  (Another reason why I don't filter instances by 
state!)

I haven't actually found where metadata caching is implemented, although the 
constructor of InstanceMetadata documents restrictions that really only make 
sense if it is.  Anyone know where it is cached?

In terms of information exposed: An alternative would be to try to connect to 
every IP in the subnet we are assigned; this blueprint can be seen as an 
optimization on that (to avoid DDOS-ing the public clouds).  So I've tried to 
expose only the information that enables directed scanning: availability zone, 
reservation id, security groups, network ids  labels  cidrs  IPs [example 
below].  A naive implementation will just try every peer; a smarter 
implementation might check the security groups to try to filter it, or the zone 
information to try to connect to nearby peers first.  Note that I don't expose 
e.g. the instance state: if you want to know whether a node is up, you have to 
try connecting to it.  I don't believe any of this information is at all 
sensitive, particularly not to instances in the same project.

On external agents doing the configuration: yes, they could put this into user 
defined metadata, but then we're tied to a configuration system.  We have to 
get 20 configuration systems to agree on a common format (Heat, Puppet, Chef, 
Ansible, SaltStack, Vagrant, Fabric, all the home-grown systems!)  It also 
makes it hard to launch instances concurrently (because you want node #2 to 
have the metadata for node #1, so you have to wait for 

Re: [openstack-dev] [Nova] bp proposal: discovery of peer instances through metadata service

[Murray, Paul (HP Cloud Services)]

Hi Justin,

My though process is to go back to basics. To perform discovery there is no 
getting away from the fact that you have to start with a well-known address 
that your peers can access on the network. The second part is a 
service/protocol accessible at that address that can perform the discovery. So 
the questions are: what well-known addresses can I reach? And is that a 
suitable place to implement the service/protocol.

The metadata service is different to the others in that it can be accessed 
without credentials (correct me if I'm wrong), so it is the only possibility 
out of the openstack services if you do not want to have credentials on the 
peer instances. If that is not the case then the other services are options. 
All services require security groups and/or networks to be configured 
appropriately to access them.

(Yes, the question can all instances access the same metadata service did 
really mean are they all local. Sorry for being unclear. But I think your 
answer is yes, they are, right?)

Implementing the peer discovery in the instances themselves requires some kind 
of multicast or knowing a list of addresses to try. In both cases either the 
actual addresses or some name resolved through a naming service would do. 
Whatever is starting your instances does have access to at least nova, so it 
can find out if there are any running instances and what their addresses are. 
These could be used as the addresses they try first. These are the way that 
internet p2p services work and they work in the cloud.

So there are options. The metadata service is a good place in terms of 
accessibility, but may not be for other reasons. In particular, the lack of 
credentials relates to the fact it is only allowed to see its own information. 
Making that more dynamic and including information about other things in the 
system might change the security model slightly. Secondly, is it the purpose of 
the metadata server to do this job? That's more a matter of choice.

Personally, I think no, this is not the right place.

Paul.



From: Justin Santa Barbara [mailto:jus...@fathomdb.com]
Sent: 24 January 2014 21:01
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] bp proposal: discovery of peer instances 
through metadata service

Murray, Paul (HP Cloud Services)  wrote:

Multicast is not generally used over the internet, so the comment about 
removing multicast is not really justified, and any of the approaches that work 
there could be used.

I think multicast/broadcast is commonly used 'behind the firewall', but I'm 
happy to hear of any other alternatives that you would recommend - particularly 
if they can work on the cloud!


I agree that the metadata service is a sensible alternative. Do you imagine 
your instances all having access to the same metadata service? Is there 
something more generic and not tied to the architecture of a single openstack 
deployment?


Not sure I understand - doesn't every Nova instance has access to the metadata 
service, and they all connect to the same back-end database?  Has anyone not 
deployed the metadata service?  It is not cross-region / cross-provider - is 
that what you mean?  In terms of implementation 
(https://review.openstack.org/#/c/68825/) it is supposed to be the same as if 
you had done a list-instances call on the API provider.  I know there's been 
talk of federation here; when this happens it would be awesome to have a 
cross-provider view (optionally, probably).

Although this is a simple example, it is also the first of quite a lot of 
useful primitives that are commonly provided by configuration services. As it 
is possible to do what you want by other means (including using an 
implementation that has multicast within subnets - I'm sure neutron does 
actually have this), it seems that this makes less of a special case and rather 
a requirement for a more general notification service?

I don't see any other solution offering as easy a solution for users (either 
the developer of the application or the person that launches the instances).  
If every instance had an automatic keystone token/trust with read-only access 
to its own project, that would be great.  If Heat intercepted every Nova call 
and added metadata, that would be great.  If Marconi offered every instance a 
'broadcast' queue where it could reach all its peers, and we had a Keystone 
trust for that, that would be great.  But, those are all 12 month projects, and 
even if you built them and they were awesome they still wouldn't get deployed 
on all the major clouds, so I _still_ couldn't rely on them as an application 
developer.

My hope is to find something that every cloud can be comfortable deploying, 
that solves discovery just as broadcast/multicast solves it on typical LANs.  
It may be that anything other than IP addresses will make e.g. HP public cloud 
uncomfortable; if so then I'll tweak it to just be IPs.  Finding

[openstack-dev] [Nova][Scheduler] Will the Scheuler use Nova Objects?

[Murray, Paul (HP Cloud Services)]

Hi,

I have heard a couple of conflicting comments about the scheduler and nova 
objects that I would like to clear up. In one scheduler/gantt meeting, Gary 
Kotton offered to convert the scheduler to use Nova objects. In another I heard 
that with the creation of Gantt, the scheduler would avoid using any Nova 
specific features including Nova objects.

I can see that these things are evolving at the same time, so it makes sense 
that plans or opinions might change. But I am at a point where it would be nice 
to know.

Which way should this go?

Paul.

Paul Murray
HP Cloud Services
+44 117 312 9309

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] do nova objects work for plugins?

[Murray, Paul (HP Cloud Services)]

I was looking through Nova Objects with a view to creating an extensible object 
that can be used by writers of plugins to include data generated by the plugin 
(others have also done the same e.g. https://review.openstack.org/#/c/65826/ ) 
On the way I noticed what I think is a bug in Nova Objects serialization (but 
might be considered by design by some - see: 
https://bugs.launchpad.net/nova/+bug/1275675). Basically, if object A has 
object B as a child, and deserialization finds object B to be an unrecognized 
version, it will try to back port the object A to the version number of object 
B.

Now this is not a problem if the version of A is always bumped when the version 
of B changes. If the A and B versions are always deployed together, because 
they are revised and built together, then A will always be the one that is 
found to be incompatible first and in back porting it will always know what 
version its child should be. If that is not the way things are meant to work 
then there is a problem (I think).

Going back to the extensible object, what I would like to be able to do is 
allow the writer of a plugin to implement a nova object for data specific to 
that plugin, so that it can be communicated by Nova. (For example, a resource 
plugin on the compute node generates resource specific data that is passed to 
the scheduler, where another plugin consumes it). This object will be 
communicated as a child of another object (e.g. the compute_node). It would be 
useful if the plugins at each end benefit from the same version handling that 
nova does itself.

It is not reasonable to bump the version of the compute_node when new external 
plugin is developed. So currently the versioning seems too rigid to implement 
extensible/pluggable objects this way.

A reasonable alternative might be for all objects to be deserialized 
individually within a tree data structure, but I'm not sure what might happen 
to parent/child compatability without some careful tracking.

Another might be to say that nova objects are for nova use only and that's just 
tough for plugin writers!

Thoughts?

Paul



Paul Murray
HP Cloud Services
+44 117 312 9309

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] do nova objects work for plugins?

[Murray, Paul (HP Cloud Services)]

Hi Dan,

Yes, we did plan to do something simple with a DictOfStringsField, I was trying 
to see if I could do the more complicated thing. BTW the simple version for 
extra_resources is here: https://review.openstack.org/#/c/66959/ 

It will be good to go through any remaining problems next week, but I think I 
can get on with it.

Paul.

-Original Message-
From: Dan Smith [mailto:d...@danplanet.com] 
Sent: 03 February 2014 20:59
To: Murray, Paul (HP Cloud Services); OpenStack Development Mailing List (not 
for usage questions)
Subject: Re: [Nova] do nova objects work for plugins?

 Basically, if object A has object B as a child, and deserialization 
 finds object B to be an unrecognized version, it will try to back port 
 the object A to the version number of object B.

Right, which is why we rev the version of, say, the InstanceList when we have 
to rev Instance itself, and why we have unit tests to makes sure that happens.

 It is not reasonable to bump the version of the compute_node when new 
 external plugin is developed. So currently the versioning seems too 
 rigid to implement extensible/pluggable objects this way.

So we're talking about an out-of-tree closed-source plugin, right? IMHO, Nova's 
versioning infrastructure is in place to make Nova able to handle upgrades; 
adding requirements for supporting out-of-tree plugins wouldn't be high on my 
priority list.

 A reasonable alternative might be for all objects to be deserialized 
 individually within a tree data structure, but I'm not sure what might 
 happen to parent/child compatability without some careful tracking.

I think it would probably be possible to make the deserializer specify the 
object and version it tripped over when passing the whole thing back to 
conductor to be backleveled. That seems reasonably useful to Nova itself.

 Another might be to say that nova objects are for nova use only and 
 that's just tough for plugin writers!

Well, for the same reason we don't provide a stable virt driver API (among 
other things) I don't think we need to be overly concerned with allowing 
arbitrary bolt-on code to hook in at this point.

Your concern is, I assume, allowing a resource metric plugin to shove actual 
NovaObject items into a container object of compute node metrics?
Is there some reason that we can't just coerce all of these to a 
dict-of-strings or dict-of-known-primitive-types to save all of this 
complication? I seem to recall the proposal that led us down this road being 
store/communicate arbitrary JSON blobs, but certainly there is a happy medium?

Given that the nova meetup is next week, perhaps that would be a good time to 
actually figure out a path forward?

--Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] about the bp cpu-entitlement

[Murray, Paul (HP Cloud Services)]

Hi Sahid,

This is being done by Oshrit Feder, so I'll let her answer, but I know that it 
is going to be implemented as an extensible resource (see: 
https://blueprints.launchpad.net/nova/+spec/extensible-resource-tracking) so it 
is waiting for that to be done. That blueprint is making good progress now and 
it should have more patches up this week. There is another resource example 
nearly done for network entitlement (see: 
https://blueprints.launchpad.net/nova/+spec/network-bandwidth-entitlement) 

Paul.

-Original Message-
From: sahid [mailto:sahid.ferdja...@cloudwatt.com] 
Sent: 04 February 2014 09:24
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [nova] about the bp cpu-entitlement

Greetings,

  I saw a really interesting blueprint about cpu entitlement, it will be 
targeted for icehouse-3 and I would like to get some details about the 
progress?. Does the developer need help? I can give a part of my time on it.

https://blueprints.launchpad.net/nova/+spec/cpu-entitlement

Thanks a lot,
s.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] What is the currently accepted way to do plugins

[Murray, Paul (HP Cloud Services)]

Hi All,

One of my patches has a query asking if I am using the agreed way to load 
plugins: https://review.openstack.org/#/c/71557/

I followed the same approach as filters/weights/metrics using nova.loadables. 
Was there an agreement to do it a different way? And if so, what is the agreed 
way of doing it? A pointer to an example or even documentation/wiki page would 
be appreciated.

Thanks in advance,
Paul

Paul Murray
HP Cloud Services
+44 117 312 9309

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins

[Murray, Paul (HP Cloud Services)]

In a chat with Dan Smith on IRC, he was suggesting that the important thing was 
not to use class paths in the config file. I can see that internal 
implementation should not be exposed in the config files - that way the 
implementation can change without impacting the nova users/operators.

Sandy, I'm not sure I really get the security argument. Python provides every 
means possible to inject code, not sure plugins are so different. Certainly 
agree on choosing which plugins you want to use though.

-Original Message-
From: Sandy Walsh [mailto:sandy.wa...@rackspace.com] 
Sent: 04 March 2014 17:50
To: OpenStack Development Mailing List (not for usage questions); Murray, Paul 
(HP Cloud Services)
Subject: RE: [openstack-dev] [Nova] What is the currently accepted way to do 
plugins

This brings up something that's been gnawing at me for a while now ... why use 
entry-point based loaders at all? I don't see the problem they're trying to 
solve. (I thought I got it for a while, but I was clearly fooling myself)

1. If you use the load all drivers in this category feature, that's a 
security risk since any compromised python library could hold a trojan.

2. otherwise you have to explicitly name the plugins you want (or don't want) 
anyway, so why have the extra indirection of the entry-point? Why not just name 
the desired modules directly? 

3. the real value of a loader would be to also extend/manage the python path 
... that's where the deployment pain is. Use fully qualified filename driver 
and take care of the pathing for me. Abstracting the module/class/function 
name isn't a great win. 

I don't see where the value is for the added pain (entry-point 
management/package metadata) it brings. 

CMV,

-S

From: Russell Bryant [rbry...@redhat.com]
Sent: Tuesday, March 04, 2014 1:29 PM
To: Murray, Paul (HP Cloud Services); OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Nova] What is the currently accepted way to do 
plugins

On 03/04/2014 06:27 AM, Murray, Paul (HP Cloud Services) wrote:
 One of my patches has a query asking if I am using the agreed way to 
 load plugins: https://review.openstack.org/#/c/71557/

 I followed the same approach as filters/weights/metrics using 
 nova.loadables. Was there an agreement to do it a different way? And 
 if so, what is the agreed way of doing it? A pointer to an example or 
 even documentation/wiki page would be appreciated.

The short version is entry-point based plugins using stevedore.

We should be careful though.  We need to limit what we expose as external plug 
points, even if we consider them unstable.  If we don't want it to be public, 
it may not make sense for it to be a plugin interface at all.

--
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] RFC - using Gerrit for Nova Blueprint review approval

[Murray, Paul (HP Cloud Services)]

The principle is excellent, I think there are two points/objectives worth 
keeping in mind:

1. We need an effective way to make and record the design decisions
2. We should make the whole development process easier

In my mind the point of the design review part is to agree up front something 
that should not be over-turned (or is hard to over-turn) late in patch review. 
I agree with others that a patch should not be blocked (or should be hard to 
block) because the reviewer disagrees with an agreed design decision. Perhaps 
an author can ask for a -2 or -1 to be removed if they can point out the agreed 
design decision, without having to reopen the debate. 

I also think that blueprints tend to have parts that should be agreed up front, 
like changes to apis, database migrations, or integration points in general. 
They also have parts that don't need to be agreed up front, there is no point 
in a heavyweight process for everything. Some blueprints might not need any of 
this at all. For example, a new plugin for the filter scheduler might no need a 
lot of design review, or at least, adding the design review is unlikely to ease 
the development cycle.

So, we could use the blueprint template to identify things that need to be 
agreed in the design review. These could include anything the proposer wants 
agreed up front and possibly specifics of a defined set of integration points. 
Some blueprints might have nothing to be formally agreed in design review. 
Additionally, sometimes plans change, so it should be possible to return to 
design review. Possibly the notion of a design decision could be broken out 
form a blueprint in the same way as a patch-set? maybe it only makes sense to 
do it as a whole? Certainly design decisions should be made in relation to 
other blueprints and so it should be easy to see that there are two blueprints 
making related design decisions.

The main point is that there should be an identifiable set of design decisions 
that have reviewed and agreed that can also be found.

**The reward for authors in doing this is the author can defend their patch-set 
against late objections to design decisions.**
**The reward for reviewers is they get a way to know what has been agreed in 
relation to a blueprint.**

On another point...
...sometimes I fall foul of writing code using an approach I have seen in the 
code base, only to be told it was decided not to do it that way anymore. 
Sometimes I had no way of knowing that, and exactly what has been decided, when 
it was decided, and who did the deciding has been lost. Clearly the PTL and ML 
do help out here, but it would be helpful if such things were easy to find out. 
These kinds of design decision should be reviewed and recorded.

Again, I think it is excellent that this is being addressed.

Paul.



-Original Message-
From: Sean Dague [mailto:s...@dague.net] 
Sent: 07 March 2014 12:01
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova] RFC - using Gerrit for Nova Blueprint 
review  approval

On 03/07/2014 06:30 AM, Thierry Carrez wrote:
 Sean Dague wrote:
 One of the issues that the Nova team has definitely hit is Blueprint 
 overload. At some point there were over 150 blueprints. Many of them 
 were a single sentence.

 The results of this have been that design review today is typically 
 not happening on Blueprint approval, but is instead happening once 
 the code shows up in the code review. So -1s and -2s on code review 
 are a mix of design and code review. A big part of which is that 
 design was never in any way sufficiently reviewed before the code started.

 In today's Nova meeting a new thought occurred. We already have 
 Gerrit which is good for reviewing things. It gives you detailed 
 commenting abilities, voting, and history. Instead of attempting (and 
 usually
 failing) on doing blueprint review in launchpad (or launchpad + an 
 etherpad, or launchpad + a wiki page) we could do something like follows:

 1. create bad blueprint
 2. create gerrit review with detailed proposal on the blueprint 3. 
 iterate in gerrit working towards blueprint approval 4. once approved 
 copy back the approved text into the blueprint (which should now be 
 sufficiently detailed)

 Basically blueprints would get design review, and we'd be pretty sure 
 we liked the approach before the blueprint is approved. This would 
 hopefully reduce the late design review in the code reviews that's 
 happening a lot now.

 There are plenty of niggly details that would be need to be worked 
 out

  * what's the basic text / template format of the design to be 
 reviewed (probably want a base template for folks to just keep things 
 consistent).
  * is this happening in the nova tree (somewhere in docs/ - NEP (Nova 
 Enhancement Proposals), or is it happening in a separate gerrit tree.
  * are there timelines for blueprint approval in a cycle? after which 
 point, we don't review any new items.

Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins

[Murray, Paul (HP Cloud Services)]

Reviewing this thread to come to a conclusion (for myself at least - and 
hopefully so I can document something so reviewers know why I did it)

For approach:
1. plugins should use stevedore with entry points (as stated by Russell)
2. the plugins should be explicitly selected through configuration 

For api stability:
I'm not sure there was a consensus. Personally I would write a base class for 
the plugins and document in it that the interface is unstable. Sound good?

BTW: this is one of those things that could be put in a place to make and 
record decisions (like the gerrit idea for blueprints). But now I am referring 
to another thread 
[http://lists.openstack.org/pipermail/openstack-dev/2014-March/029232.html ]

Paul.


-Original Message-
From: Sandy Walsh [mailto:sandy.wa...@rackspace.com] 
Sent: 04 March 2014 21:25
To: Murray, Paul (HP Cloud Services)
Cc: OpenStack Development Mailing List (not for usage questions); 
d...@danplanet.com
Subject: Re: [openstack-dev] [Nova] What is the currently accepted way to do 
plugins

And sorry, as to your original problem, the loadables approach is kinda messy 
since only the classes that are loaded when *that* module are loaded are used 
(vs. explicitly specifying them in a config). You may get different results 
when the flow changes.

Either entry-points or config would give reliable results.


On 03/04/2014 03:21 PM, Murray, Paul (HP Cloud Services) wrote:
 In a chat with Dan Smith on IRC, he was suggesting that the important thing 
 was not to use class paths in the config file. I can see that internal 
 implementation should not be exposed in the config files - that way the 
 implementation can change without impacting the nova users/operators.

There's plenty of easy ways to deal with that problem vs. entry points.

MyModule.get_my_plugin() ... which can point to anywhere in the module 
permanently.

Also, we don't have any of the headaches of merging setup.cfg sections (as we 
see with oslo.* integration).

 Sandy, I'm not sure I really get the security argument. Python provides every 
 means possible to inject code, not sure plugins are so different. Certainly 
 agree on choosing which plugins you want to use though.

The concern is that any compromised part of the python eco-system can get 
auto-loaded with the entry-point mechanism. Let's say Nova auto-loads all 
modules with entry-points the [foo] section. All I have to do is create a setup 
that has a [foo] section and my code is loaded.
Explicit is better than implicit.

So, assuming we don't auto-load modules ... what does the entry-point approach 
buy us?


 From: Russell Bryant [rbry...@redhat.com] We should be careful though.  
 We need to limit what we expose as external plug points, even if we consider 
 them unstable.  If we don't want it to be public, it may not make sense for 
 it to be a plugin interface at all.

I'm not sure what the concern with introducing new extension points is?
OpenStack is basically just a big bag of plugins. If it's optional, it's 
supposed to be a plugin (according to the design tenets).



 
 --
 Russell Bryant
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Hosts within two Availability Zones : possible or not ?

[Murray, Paul (HP Cloud Services)]

Hi Sylvain,

I would go with keeping AZs exclusive. It is a well-established concept even if 
it is up to providers to implement what it actually means in terms of 
isolation. Some good use cases have been presented on this topic recently, but 
for me they suggest we should develop a better concept rather than bend the 
meaning of the old one. We certainly don't have hosts in more than one AZ in HP 
Cloud and I think some of our users would be very surprised if we changed that.

Paul.

From: Khanh-Toan Tran [mailto:khanh-toan.t...@cloudwatt.com]
Sent: 03 April 2014 15:53
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] Hosts within two Availability Zones : 
possible or not ?

+1 for AZs not sharing hosts.

Because it's the only mechanism that allows us to segment the datacenter. 
Otherwise we cannot provide redundancy to client except using Region which is 
dedicated infrastructure and networked separated and anti-affinity filter which 
IMO is not pragmatic as it has tendency of abusive usage.  Why sacrificing this 
power so that users can select the types of his desired physical hosts ? The 
latter can be exposed using flavor metadata, which is a lot safer and more 
controllable than using AZs. If someone insists that we really need to let 
users choose the types of physical hosts, then I suggest creating a new hint, 
and use aggregates with it. Don't sacrifice AZ exclusivity!

Btw, there is a datacenter design called dual-room [1] which I think best fit 
for AZs to make your cloud redundant even with one datacenter.

Best regards,

Toan

[1] IBM and Cisco: Together for a World Class Data Center, Page 141. 
http://books.google.fr/books?id=DHjJAgAAQBAJpg=PA141#v=onepageqf=false



De : Sylvain Bauza [mailto:sylvain.ba...@gmail.com]
Envoyé : jeudi 3 avril 2014 15:52
À : OpenStack Development Mailing List (not for usage questions)
Objet : [openstack-dev] [Nova] Hosts within two Availability Zones : possible 
or not ?

Hi,

I'm currently trying to reproduce [1]. This bug requires to have the same host 
on two different aggregates, each one having an AZ.

IIRC, Nova API prevents hosts of being part of two distinct AZs [2], so IMHO 
this request should not be possible.
That said, there are two flaws where I can identify that no validation is done :
 - when specifying an AZ in nova.conf, the host is overriding the existing AZ 
by its own
 - when adding an host to an aggregate without AZ defined, and afterwards 
update the aggregate to add an AZ


So, I need direction. Either we consider it is not possible to share 2 AZs for 
the same host and then we need to fix the two above scenarios, or we say it's 
nice to have 2 AZs for the same host and then we both remove the validation 
check in the API and we fix the output issue reported in the original bug [1].


Your comments are welcome.
Thanks,
-Sylvain


[1] : https://bugs.launchpad.net/nova/+bug/1277230

[2] : 
https://github.com/openstack/nova/blob/9d45e9cef624a4a972c24c47c7abd57a72d74432/nova/compute/api.py#L3378
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler][metrics] Additional metrics

[Murray, Paul (HP Cloud Services)]

Hi Abbass, 

I am in the process of coding some of this now - take a look at 

https://blueprints.launchpad.net/nova/+spec/extensible-resource-tracking - now 
has a specification document attached 
https://etherpad.openstack.org/p/IcehouseNovaExtensibleSchedulerMetrics - the 
design summit session on this topic

see what you think and feel free to comment - I think it covers exactly what 
you describe.

Paul.


Paul Murray
HP Cloud Services
+44 117 312 9309

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England. The contents of this message and any attachments 
to it are confidential and may be legally privileged. If you have received this 
message in error, you should delete it from your system immediately and advise 
the sender. To any recipient of this message within HP, unless otherwise stated 
you should consider this message and attachments as HP CONFIDENTIAL.



-Original Message-
From: Lu, Lianhao [mailto:lianhao...@intel.com] 
Sent: 22 November 2013 02:03
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][scheduler][metrics] Additional metrics


Abbass MAROUNI wrote on 2013-11-21:
 Hello,
 
 I'm in the process of writing a new scheduling algorithm for openstack nova.
 I have a set of compute nodes that I'm going to filter and weigh according to 
 some metrics collected from these compute nodes.
 I saw nova.compute.resource_tracker and metrics (ram, disk and cpu) 
 that it collects from compute nodes and updates the rows corresponding to 
 compute nodes in the database.
 
 I'm planning to write some modules that will collect the new metrics 
 but I'm wondering if I need to modify the database schema by adding 
 more columns in the 'compute_nodes' table for my new metrics. Will 
 this require some modification to the compute model ? Then how can I use 
 these metrics during the scheduling process, do I fetch each compute node row 
 from the database ? Is there any easier way around this problem ?
 
 Best Regards,

There are currently some effort on this:
https://blueprints.launchpad.net/nova/+spec/utilization-aware-scheduling
https://blueprints.launchpad.net/nova/+spec/extensible-resource-tracking 

- Lianhao


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][database] Update compute_nodes table

[Murray, Paul (HP Cloud Services)]

Hi Abbass,

I guess you read the blueprint Russell referred to. I think you actually are 
saying the same - but please read steps below and tell me if they don't cover 
what you want.

This is what it will do:

1.   Add a column to the compute_nodes table for a JSON blob

2.   Add plug-in framework for additional resources in resource_tacker 
(just like filters in filter scheduler)

3.   Resource plugin classes will implement things like:

a.   Claims test method

b.  add your data here method (so it can populate the JSON blob)

4.   Additional column is available in host_state at filter scheduler

You will then be able to do any or all of the following:

1.   Add new parameters to requests in extra_specs

2.   Add new filter/weight classes as scheduler plugins

a.   Will have access to filter properties (including extra_specs)

b.  Will have access to extra resource data (from compute node)

c.   Can generate limits

3.   Add new resource classes as scheduler plugins

a.   Will have access to filter properties (including extra specs)

b.  Will have access to limits (from scheduler)

c.   Can generate extra resource data to go to scheduler

Does this match your needs?

There are also plans to change how data goes from compute nodes to scheduler 
(i.e. not through the database). This will remove the database from the 
equation. But that can be kept as a separate concern.

Paul.



From: Abbass MAROUNI [mailto:abbass.maro...@virtualscale.fr]
Sent: 03 December 2013 08:54
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [nova][database] Update compute_nodes table

I am aware of this work, in fact I reused a column (pci_stats) in the 
compute_nodes table to store a JSON blob.
I track the resource in the resource_tracker and update the column and then use 
the blob in a filter.
Maybe I should reformulate my question, How can I add a column to the table and 
use it in resource_tracker without breaking something ?

Best regards,

2013/12/2 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org


--

Message: 1
Date: Mon, 02 Dec 2013 12:06:21 -0500
From: Russell Bryant rbry...@redhat.commailto:rbry...@redhat.com
To: openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [nova][database] Update compute_nodes
table
Message-ID: 529cbe0d@redhat.commailto:529cbe0d@redhat.com
Content-Type: text/plain; charset=ISO-8859-1

On 12/02/2013 11:47 AM, Abbass MAROUNI wrote:
 Hello,

 I'm looking for way to a add new attribute to the compute nodes by
  adding a column to the compute_nodes table in nova database in order to
 track a metric on the compute nodes and use later it in nova-scheduler.

 I checked the  sqlalchemy/migrate_repo/versions and thought about adding
 my own upgrade then sync using nova-manage db sync.

 My question is :
 What is the process of upgrading a table in the database ? Do I have to
 modify or add a new variable in some class in order to associate the
 newly added column with a variable that I can use ?

Don't add this.  :-)

There is work in progress to just have a column with a json blob in it
for additional metadata like this.

https://blueprints.launchpad.net/nova/+spec/extensible-resource-tracking
https://wiki.openstack.org/wiki/ExtensibleResourceTracking

--
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] New DB column or new DB table?

[Murray, Paul (HP Cloud Services)]

Hi All,

I would like to chip in with something from the side here (sorry to stretch the 
discussion out).

I was looking for a mechanism to do something like this in the context of this 
blueprint on network aware scheduling: 
https://blueprints.launchpad.net/nova/+spec/network-bandwidth-entitlement 
Essentially the problem is that I need to add network bandwidth resource 
allocation information just like vcpu, memory and disk space already has. I 
could hard code this just as they are, but I can also think of a couple of 
others we would like to add that may be more specific to a given installation. 
So I could do with a generic way to feed this information back from the compute 
node to the scheduler just like this.

However, my use case is not the same - it is not meant to be for 
monitored/statistical utilization info. But I would like a similar mechanism to 
allow the scheduler to keep track of more general / extensible resource 
allocation.

Do you have any thoughts on that? Again, don't mean to deflect the discussion - 
just I have another use case.

Paul.


-Original Message-
From: Sean Dague [mailto:s...@dague.net]
Sent: 18 July 2013 12:05
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Nova] New DB column or new DB table?

On 07/17/2013 10:54 PM, Lu, Lianhao wrote:
 Hi fellows,

 Currently we're implementing the BP
https://blueprints.launchpad.net/nova/+spec/utilization-aware-scheduling.
The main idea is to have an extensible plugin framework on nova-compute 
where every plugin can get different metrics(e.g. CPU utilization, 
memory cache utilization, network bandwidth, etc.) to store into the 
DB, and the nova- scheduler will use that data from DB for scheduling decision.

 Currently we adds a new table to store all the metric data and have 
 nova-
scheduler join loads the new table with the compute_nodes table to get 
all the data(https://review.openstack.org/35759). Someone is concerning 
about the performance penalty of the join load operation when there are 
many metrics data stored in the DB for every single compute node. Don 
suggested adding a new column in the current compute_nodes table in DB, 
and put all metric data into a dictionary key/value format and store 
the json encoded string of the dictionary into that new column in DB.

 I'm just wondering which way has less performance impact, join load 
 with a
new table with quite a lot of rows, or json encode/decode a dictionary 
with a lot of key/value pairs?

 Thanks,
 -Lianhao

I'm really confused. Why are we talking about collecting host metrics 
in nova when we've got a whole project to do that in ceilometer? I 
think utilization based scheduling would be a great thing, but it 
really out to be interfacing with ceilometer to get that data. Storing 
it again in nova (or even worse collecting it a second time in nova) seems 
like the wrong direction.

I think there was an equiv patch series at the end of Grizzly that was 
pushed out for the same reasons.

If there is a reason ceilometer can't be used in this case, we should 
have that discussion here on the list. Because my initial reading of 
this blueprint and the code patches is that it partially duplicates 
ceilometer function, which we definitely don't want to do. Would be happy to 
be proved wrong on that.

   -Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] New DB column or new DB table?

[Murray, Paul (HP Cloud Services)]

Hi Jay, Lianhao, All,

Sorry if this comes out of order - for some reason I am not receiving the 
messages so I'm cut-and-pasting from the archive :( 

I think I might mean something closer to Brian's blue print (now I've seen it) 
https://blueprints.launchpad.net/nova/+spec/heterogeneous-instance-types 

Really I want to do resource management the way vcpu, memory and disk do. The 
scheduler chooses where to place instances according to an understanding of the 
available and free resources (and updates that when scheduling multiple 
instances, as in the consume_from_instance method of 
nova.scheduler.host_manager.HostState). Likewise, the compute node checks (in 
the test method of nova.compute.claims.Claim ) that they are available before 
accepting an instance. When the instance is created it reports back the usage 
to the database via the resource tracker. This is actually accounting what has 
been allocated, not an on-going measure of what is being used. 

Extra specs can certainly be used, but that does not provide the feedback loop 
between the compute nodes and the scheduler necessary to do the accounting of 
resource consumption.

What I would need for a generic way to do this is plugins at the compute node, 
a way to pass arbitrary resource consumption information back through the 
database, and plugins at the scheduler. So I am going beyond what is described 
here but the basic mechanisms are the same. The alternative is to code in each 
new resource we want to manage (which may not be that many really - but they 
may not be there for all installations).

Interestingly the 
https://blueprints.launchpad.net/nova/+spec/generic-host-state-for-scheduler 
blueprint (referenced in the patch) does talk about going to ceilometer. And 
that does seem to make sense to me. 

BTW, I'm getting all the other emails - just not this thread!

Bemused...
Paul


On 07/18/2013 10:44 AM, Murray, Paul (HP Cloud Services) wrote:
 Hi All,

 I would like to chip in with something from the side here (sorry to stretch 
 the discussion out).

 I was looking for a mechanism to do something like this in the context of 
 this blueprint on network aware scheduling: 
 https://blueprints.launchpad.net/nova/+spec/network-bandwidth-entitlement 
 Essentially the problem is that I need to add network bandwidth resource 
 allocation information just like vcpu, memory and disk space already has. I 
 could hard code this just as they are, but I can also think of a couple of 
 others we would like to add that may be more specific to a given 
 installation. So I could do with a generic way to feed this information back 
 from the compute node to the scheduler just like this.

 However, my use case is not the same - it is not meant to be for 
 monitored/statistical utilization info. But I would like a similar mechanism 
 to allow the scheduler to keep track of more general / extensible resource 
 allocation.

How is that a different use case from Lianhao's? You mean instead of 
collected usage metrics you want to allocate based on the value of a 
transient statistic like current network bandwidth utilisation?

 Do you have any thoughts on that? Again, don't mean to deflect the discussion 
 - just I have another use case.

I tend to agree with both Brian and Sean on this. I agree with Sean in 
that it seems duplicative to store compute_node_resources in the Nova 
database when a simple REST call to Ceilometer would avoid the 
duplication. And I agree with Brian that the extra_specs scheduler 
filters seem like they would fit the check a current bandwidth 
statistic type use case you describe above, Paul.

Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?)

[Murray, Paul (HP Cloud Services)]

Hi Sean,

Do you think the existing static allocators should be migrated to going through 
ceilometer - or do you see that as different? Ignoring backward compatibility.

The reason I ask is I want to extend the static allocators to include a couple 
more. These plugins are the way I would have done it. Which way do you think 
that should be done?

Paul.

-Original Message-
From: Sean Dague [mailto:s...@dague.net] 
Sent: 19 July 2013 12:04
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics 
collector for scheduling (was: New DB column or new DB table?)

On 07/19/2013 06:18 AM, Day, Phil wrote:
 Ceilometer is a great project for taking metrics available in Nova and other 
 systems and making them available for use by Operations, Billing, Monitoring, 
 etc - and clearly we should try and avoid having multiple collectors of the 
 same data.

 But making the Nova scheduler dependent on Ceilometer seems to be the wrong 
 way round to me - scheduling is such a fundamental operation that I want Nova 
 to be self sufficient in this regard.   In particular I don't want the 
 availability of my core compute platform to be constrained by the 
 availability of my (still evolving) monitoring system.

 If Ceilometer can be fed from the data used by the Nova scheduler then that's 
 a good plus - but not the other way round.

I assume it would gracefully degrade to the existing static allocators if 
something went wrong. If not, well that would be very bad.

Ceilometer is an integrated project in Havana. Utilization based scheduling 
would be a new feature. I'm not sure why we think that duplicating the metrics 
collectors in new code would be less buggy than working with Ceilometer. Nova 
depends on external projects all the time.

If we have a concern about robustness here, we should be working as an overall 
project to address that.

-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling

[Murray, Paul (HP Cloud Services)]

If we agree that something like capabilities should go through Nova, what do 
you suggest should be done with the change that sparked this debate: 
https://review.openstack.org/#/c/35760/ 

I would be happy to use it or a modified version.

Paul.

-Original Message-
From: Sean Dague [mailto:s...@dague.net] 
Sent: 19 July 2013 14:28
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics 
collector for scheduling

On 07/19/2013 08:30 AM, Andrew Laski wrote:
 On 07/19/13 at 12:08pm, Murray, Paul (HP Cloud Services) wrote:
 Hi Sean,

 Do you think the existing static allocators should be migrated to 
 going through ceilometer - or do you see that as different? Ignoring 
 backward compatibility.

 It makes sense to keep some things in Nova, in order to handle the 
 graceful degradation needed if Ceilometer couldn't be reached.  I see 
 the line as something like capabilities should be handled by Nova, 
 memory free, vcpus available, etc... and utilization metrics handled 
 by Ceilometer.

Yes, that makes sense to me. I'd be happy with that.

-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev