Re: [openstack-dev] [nova] Resource tracker

2014-10-07 Thread Daniel P. Berrange
On Mon, Oct 06, 2014 at 02:55:20PM -0700, Joe Gordon wrote:
 On Mon, Oct 6, 2014 at 6:03 AM, Gary Kotton gkot...@vmware.com wrote:
 
   Hi,
  At the moment the resource tracker in Nova ignores that statistics that
  are returned by the hypervisor and it calculates the values on its own. Not
  only is this highly error prone but it is also very costly – all of the
  resources on the host are read from the database. Not only the fact that we
  are doing something very costly is troubling, the fact that we are over
  calculating resources used by the hypervisor is also an issue. In my
  opinion this leads us to not fully utilize hosts at our disposal. I have a
  number of concerns with this approach and would like to know why we are not
  using the actual resource reported by the hypervisor.
  The reason for asking this is that I have added a patch which uses the
  actual hypervisor resources returned and it lead to a discussion on the
  particular review (https://review.openstack.org/126237).
 
 
 So it sounds like you have mentioned two concerns here:
 
 1. The current method to calculate hypervisor usage is expensive in terms
 of database access.
 2. Nova ignores that statistics that are returned by the hypervisor and
 uses its own calculations.
 
 
 To #1, maybe we can doing something better, optimize the query, cache the
 result etc. As for #2 nova intentionally doesn't use the hypervisor
 statistics for a few reasons:
 
 * Make scheduling more deterministic, make it easier to reproduce issues
 etc.
 * Things like memory ballooning and thin provisioning in general, mean that
 the hypervisor is not reporting how much of the resources can be allocated
 but rather how much are currently in use (This behavior can vary from
 hypervisor to hypervisor today AFAIK -- which makes things confusing). So
 if I don't want to over subscribe RAM, and the hypervisor is using memory
 ballooning, the hypervisor statistics are mostly useless. I am sure there
 are more complex schemes that we can come up with that allow us to factor
 in the properties of thin provisioning, but is the extra complexity worth
 it?

That is just an example of problems with the way Nova virt drivers
/currently/ report usage to the schedular. It is easily within the
realm of possibility for the virt drivers to be changed so that they
report stats which take into account things like ballooning and thin
provisioning so that we don't oversubscribe. Ignoring the hypervisor
stats entirely and re-doing the calculations in the resource tracker
code is just a crude workaround really. It is just swapping one set
of problems for a new set of problems. 

 That being said I am fine with discussing in a spec the idea of adding an
 option to use the hypervisor reported statistics, as long as it is off by
 default.

I'm against the idea of adding config options to switch between multiple
codepaths because it is just punting the problem to the admins who are
in an even worse position to decide what is best. It is saying would you
rather your cloud have bug A or have bug B. We should be fixing the data
the hypervisors report so that the resource tracker doesn't have to ignore
them, and give the admins something which just works and avoid having to
choose between 2 differently broken options.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Resource tracker

2014-10-07 Thread Vishvananda Ishaya

On Oct 7, 2014, at 6:21 AM, Daniel P. Berrange berra...@redhat.com wrote:

 On Mon, Oct 06, 2014 at 02:55:20PM -0700, Joe Gordon wrote:
 On Mon, Oct 6, 2014 at 6:03 AM, Gary Kotton gkot...@vmware.com wrote:
 
 Hi,
 At the moment the resource tracker in Nova ignores that statistics that
 are returned by the hypervisor and it calculates the values on its own. Not
 only is this highly error prone but it is also very costly – all of the
 resources on the host are read from the database. Not only the fact that we
 are doing something very costly is troubling, the fact that we are over
 calculating resources used by the hypervisor is also an issue. In my
 opinion this leads us to not fully utilize hosts at our disposal. I have a
 number of concerns with this approach and would like to know why we are not
 using the actual resource reported by the hypervisor.
 The reason for asking this is that I have added a patch which uses the
 actual hypervisor resources returned and it lead to a discussion on the
 particular review (https://review.openstack.org/126237).
 
 
 So it sounds like you have mentioned two concerns here:
 
 1. The current method to calculate hypervisor usage is expensive in terms
 of database access.
 2. Nova ignores that statistics that are returned by the hypervisor and
 uses its own calculations.
 
 
 To #1, maybe we can doing something better, optimize the query, cache the
 result etc. As for #2 nova intentionally doesn't use the hypervisor
 statistics for a few reasons:
 
 * Make scheduling more deterministic, make it easier to reproduce issues
 etc.
 * Things like memory ballooning and thin provisioning in general, mean that
 the hypervisor is not reporting how much of the resources can be allocated
 but rather how much are currently in use (This behavior can vary from
 hypervisor to hypervisor today AFAIK -- which makes things confusing). So
 if I don't want to over subscribe RAM, and the hypervisor is using memory
 ballooning, the hypervisor statistics are mostly useless. I am sure there
 are more complex schemes that we can come up with that allow us to factor
 in the properties of thin provisioning, but is the extra complexity worth
 it?
 
 That is just an example of problems with the way Nova virt drivers
 /currently/ report usage to the schedular. It is easily within the
 realm of possibility for the virt drivers to be changed so that they
 report stats which take into account things like ballooning and thin
 provisioning so that we don't oversubscribe. Ignoring the hypervisor
 stats entirely and re-doing the calculations in the resource tracker
 code is just a crude workaround really. It is just swapping one set
 of problems for a new set of problems.

+1 lets make the hypervisors report detailed enough information that we
can do it without having to recalculate.

Vish

 
 That being said I am fine with discussing in a spec the idea of adding an
 option to use the hypervisor reported statistics, as long as it is off by
 default.
 
 I'm against the idea of adding config options to switch between multiple
 codepaths because it is just punting the problem to the admins who are
 in an even worse position to decide what is best. It is saying would you
 rather your cloud have bug A or have bug B. We should be fixing the data
 the hypervisors report so that the resource tracker doesn't have to ignore
 them, and give the admins something which just works and avoid having to
 choose between 2 differently broken options.
 
 
 Regards,
 Daniel
 -- 
 |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
 |: http://libvirt.org  -o- http://virt-manager.org :|
 |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
 |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Resource tracker

2014-10-07 Thread Joe Gordon
On Tue, Oct 7, 2014 at 10:56 AM, Vishvananda Ishaya vishvana...@gmail.com
wrote:


 On Oct 7, 2014, at 6:21 AM, Daniel P. Berrange berra...@redhat.com
 wrote:

  On Mon, Oct 06, 2014 at 02:55:20PM -0700, Joe Gordon wrote:
  On Mon, Oct 6, 2014 at 6:03 AM, Gary Kotton gkot...@vmware.com wrote:
 
  Hi,
  At the moment the resource tracker in Nova ignores that statistics that
  are returned by the hypervisor and it calculates the values on its
 own. Not
  only is this highly error prone but it is also very costly – all of the
  resources on the host are read from the database. Not only the fact
 that we
  are doing something very costly is troubling, the fact that we are over
  calculating resources used by the hypervisor is also an issue. In my
  opinion this leads us to not fully utilize hosts at our disposal. I
 have a
  number of concerns with this approach and would like to know why we
 are not
  using the actual resource reported by the hypervisor.
  The reason for asking this is that I have added a patch which uses the
  actual hypervisor resources returned and it lead to a discussion on the
  particular review (https://review.openstack.org/126237).
 
 
  So it sounds like you have mentioned two concerns here:
 
  1. The current method to calculate hypervisor usage is expensive in
 terms
  of database access.
  2. Nova ignores that statistics that are returned by the hypervisor and
  uses its own calculations.
 
 
  To #1, maybe we can doing something better, optimize the query, cache
 the
  result etc. As for #2 nova intentionally doesn't use the hypervisor
  statistics for a few reasons:
 
  * Make scheduling more deterministic, make it easier to reproduce issues
  etc.
  * Things like memory ballooning and thin provisioning in general, mean
 that
  the hypervisor is not reporting how much of the resources can be
 allocated
  but rather how much are currently in use (This behavior can vary from
  hypervisor to hypervisor today AFAIK -- which makes things confusing).
 So
  if I don't want to over subscribe RAM, and the hypervisor is using
 memory
  ballooning, the hypervisor statistics are mostly useless. I am sure
 there
  are more complex schemes that we can come up with that allow us to
 factor
  in the properties of thin provisioning, but is the extra complexity
 worth
  it?
 
  That is just an example of problems with the way Nova virt drivers
  /currently/ report usage to the schedular. It is easily within the
  realm of possibility for the virt drivers to be changed so that they
  report stats which take into account things like ballooning and thin
  provisioning so that we don't oversubscribe. Ignoring the hypervisor
  stats entirely and re-doing the calculations in the resource tracker
  code is just a crude workaround really. It is just swapping one set
  of problems for a new set of problems.


I agree, lets make reported hypervisor stats actually useful for
scheduling. This would mean we can have fewer config options (currently the
operator has to set aside resources for the underlying OS via a config
option).



 +1 lets make the hypervisors report detailed enough information that we
 can do it without having to recalculate.


Do we have any idea of how expensive recalculating this information is?



 Vish

 
  That being said I am fine with discussing in a spec the idea of adding
 an
  option to use the hypervisor reported statistics, as long as it is off
 by
  default.
 
  I'm against the idea of adding config options to switch between multiple
  codepaths because it is just punting the problem to the admins who are
  in an even worse position to decide what is best. It is saying would you
  rather your cloud have bug A or have bug B. We should be fixing the data
  the hypervisors report so that the resource tracker doesn't have to
 ignore
  them, and give the admins something which just works and avoid having to
  choose between 2 differently broken options.


 
  Regards,
  Daniel
  --
  |: http://berrange.com  -o-
 http://www.flickr.com/photos/dberrange/ :|
  |: http://libvirt.org  -o-
 http://virt-manager.org :|
  |: http://autobuild.org   -o-
 http://search.cpan.org/~danberr/ :|
  |: http://entangle-photo.org   -o-
 http://live.gnome.org/gtk-vnc :|
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Resource tracker

2014-10-06 Thread Daniel P. Berrange
On Mon, Oct 06, 2014 at 01:03:01PM +, Gary Kotton wrote:
 Hi,
 At the moment the resource tracker in Nova ignores that statistics that
 are returned by the hypervisor and it calculates the values on its own.
 Not only is this highly error prone but it is also very costly - all of
 the resources on the host are read from the database. Not only the fact
 that we are doing something very costly is troubling, the fact that we
 are over calculating resources used by the hypervisor is also an issue. 
 In my opinion this leads us to not fully utilize hosts at our disposal.
 I have a number of concerns with this approach and would like to know
 why we are not using the actual resource reported by the hypervisor.
 The reason for asking this is that I have added a patch which uses the
 actual hypervisor resources returned and it lead to a discussion on the
 particular review (https://review.openstack.org/126237).

If i'm interpreting git history correctly, this behaviour was (re-)introduced
in this commit:

  commit 8e851409f3a8a345ec954a880c81232fbf9e27b4
  Author: Brian Elliott brian.elli...@rackspace.com
  Date:   Fri Sep 14 15:17:07 2012 +

Fix bugs in resource tracker and cleanup

Fixes bugs in resource tracker:
* Handle disk oversubscription
* Handle suspended/powered off instances

The usage model is changed to the old style that is
based on actual instance usage on a compute host.
(Not the current point in time of the hypervisor's
 reported host stats)

There is now a 'limits' filter property that can be passed from
the scheduler to the compute node to indicate that
oversubscription of resources is desired:

The 'limits' filter property is a dict with the following possible
keys:

* memory_mb - Specifies the memory ceiling for the compute node.
* disk_gb - Specifies the disk space ceiling for the compute node.
* vcpu - Specifies the max number of vcpus for the compute node.

There is also some general cleanup and additional unit tests in
an attempt to simplify down this function.

bug 1048842
bug 1052157

Change-Id: I6ee851b8c03234a78a64d9f5c494dfc7059cdda4

Unfortunately that commit message isn't very informative as to why this
change was made, and the bugs don't seem to have any real detail either.

Perhaps Brian remembers himself ?

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Resource tracker

2014-10-06 Thread Joe Gordon
On Mon, Oct 6, 2014 at 6:03 AM, Gary Kotton gkot...@vmware.com wrote:

  Hi,
 At the moment the resource tracker in Nova ignores that statistics that
 are returned by the hypervisor and it calculates the values on its own. Not
 only is this highly error prone but it is also very costly – all of the
 resources on the host are read from the database. Not only the fact that we
 are doing something very costly is troubling, the fact that we are over
 calculating resources used by the hypervisor is also an issue. In my
 opinion this leads us to not fully utilize hosts at our disposal. I have a
 number of concerns with this approach and would like to know why we are not
 using the actual resource reported by the hypervisor.
 The reason for asking this is that I have added a patch which uses the
 actual hypervisor resources returned and it lead to a discussion on the
 particular review (https://review.openstack.org/126237).


So it sounds like you have mentioned two concerns here:

1. The current method to calculate hypervisor usage is expensive in terms
of database access.
2. Nova ignores that statistics that are returned by the hypervisor and
uses its own calculations.


To #1, maybe we can doing something better, optimize the query, cache the
result etc. As for #2 nova intentionally doesn't use the hypervisor
statistics for a few reasons:

* Make scheduling more deterministic, make it easier to reproduce issues
etc.
* Things like memory ballooning and thin provisioning in general, mean that
the hypervisor is not reporting how much of the resources can be allocated
but rather how much are currently in use (This behavior can vary from
hypervisor to hypervisor today AFAIK -- which makes things confusing). So
if I don't want to over subscribe RAM, and the hypervisor is using memory
ballooning, the hypervisor statistics are mostly useless. I am sure there
are more complex schemes that we can come up with that allow us to factor
in the properties of thin provisioning, but is the extra complexity worth
it?

That being said I am fine with discussing in a spec the idea of adding an
option to use the hypervisor reported statistics, as long as it is off by
default.




 Thanks
 Gary

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev