Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
Sandy Walsh wrote on 2013-07-19: > > > On 07/19/2013 09:47 AM, Day, Phil wrote: >>> -Original Message- >>> From: Sean Dague [mailto:s...@dague.net] >>> Sent: 19 July 2013 12:04 >>> To: OpenStack Development Mailing List >>> Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics >>> collector for scheduling (was: New DB column or new DB table?) >>> >>> On 07/19/2013 06:18 AM, Day, Phil wrote: >>>> Ceilometer is a great project for taking metrics available in Nova and >>>> other >>> systems and making them available for use by Operations, Billing, >>> Monitoring, etc - and clearly we should try and avoid having multiple >>> collectors of the same data. >>>> >>>> But making the Nova scheduler dependent on Ceilometer seems to be the >>> wrong way round to me - scheduling is such a fundamental operation that I >>> want Nova to be self sufficient in this regard. In particular I don't >>> want the >>> availability of my core compute platform to be constrained by the >>> availability >>> of my (still evolving) monitoring system. >>>> >>>> If Ceilometer can be fed from the data used by the Nova scheduler then > that's >>> a good plus - but not the other way round. >>> >>> I assume it would gracefully degrade to the existing static allocators if >>> something went wrong. If not, well that would be very bad. >>> >>> Ceilometer is an integrated project in Havana. Utilization based >>> scheduling would be a new feature. I'm not sure why we think that >>> duplicating the metrics collectors in new code would be less buggy >>> than working with Ceilometer. Nova depends on external projects all >>> the time. >>> >>> If we have a concern about robustness here, we should be working as an >>> overall project to address that. >>> >>> -Sean >>> >> Just to be cleat its about a lot more than just robustness in the code - its >> the > whole architectural pattern of putting Ceilometer at the centre of Nova > scheduling that concerns me. >> >> As I understand it Celiometer can collect metrics from more than one copy of > Nova - which is good; I want to run multiple independent copies in different > regions and I want to have all of my monitoring data going back to one place. > However that doesn't mean that I now also want all of those independent copies > of Nova depending on that central monitoring infrastructure for something as > basic as scheduling. (I don't want to stop anyone that does either - but I > don't > see why I should be forced down that route). >> >> The original change that sparked this debate came not from anything to do > with utilisation based scheduling, but the pretty basic and simple desire to > add > new types of consumable resource counters into the scheduler logic in a more > general way that having to make a DB schema change. This was generally > agreed to be a good thing, and it pains me to see that valuable work now > blocked > on what seems to be turning into an strategic discussion around the role of > Ceilometer (Is it a monitoring tool or a fundamental metric bus, etc). >> >> At the point where Ceilomter can be shown to replace the current scheduler > resource mgmt code in Nova, then we should be talking about switching to it - > but in the meantime why can't we continue to have incremental improvements > in the current Nova code ? > > +1 +1 We can keep discussion to determine R&R for Nova and Ceilometer. Can we have a decision to make so we can move forward to have incremental improvements in Nova? Thanks. -- Shane > >> >> Cheers >> Phil >> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
On Fri, Jul 19 2013, Sean Dague wrote: > I assume it would gracefully degrade to the existing static allocators if > something went wrong. If not, well that would be very bad. > > Ceilometer is an integrated project in Havana. Utilization based scheduling > would be a new feature. I'm not sure why we think that duplicating the > metrics collectors in new code would be less buggy than working with > Ceilometer. Nova depends on external projects all the time. > > If we have a concern about robustness here, we should be working as an > overall project to address that. +1 -- Julien Danjou -- Free Software hacker - freelance consultant -- http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
On 07/19/2013 12:30 PM, Sean Dague wrote: > On 07/19/2013 10:37 AM, Murray, Paul (HP Cloud Services) wrote: >> If we agree that "something like capabilities" should go through Nova, >> what do you suggest should be done with the change that sparked this >> debate: https://review.openstack.org/#/c/35760/ >> >> I would be happy to use it or a modified version. > > CPU sys, user, idle, iowait time isn't capabilities though. That's a > dynamically changing value. I also think the current approach where this > is point in time sampling, because we only keep a single value, is going > to cause some oddly pathologic behavior if you try to use it as > scheduling criteria. > > I'd really appreciate the views of more nova core folks on this thread, > as it looks like these blueprints have seen pretty minimal code review > at this point. H3 isn't that far away, and there is a lot of high > priority things ahead of this, and only so much coffee and review time > in a day. You really need to have a moving window average of these meters in order to have anything sensible. Also, some sort of view into the pipeline of scheduler requests (what's coming up?) Capabilities are only really used in the host filtering phase. The host weighing phase is where these measurements would be applied. > -Sean > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
I think that current approach of Scheduler is not scalable and not flexible. if we add key/value this will make our scheduler flexible but with tons of hacks and less scalable. We will get a tons of problems even on small 1k nodes cloud. We found another approach (just remove DB) this will solve all our problems. There is another thread with link to google doc, with large description. https://docs.google.com/document/d/1_DRv7it_mwalEZzLy5WO92TJcummpmWL4NWsWf0UWiQ/edit# 19.07.2013, в 19:30, Sean Dague написал(а): > On 07/19/2013 10:37 AM, Murray, Paul (HP Cloud Services) wrote: >> If we agree that "something like capabilities" should go through Nova, what >> do you suggest should be done with the change that sparked this debate: >> https://review.openstack.org/#/c/35760/ >> >> I would be happy to use it or a modified version. > > CPU sys, user, idle, iowait time isn't capabilities though. That's a > dynamically changing value. I also think the current approach where this is > point in time sampling, because we only keep a single value, is going to > cause some oddly pathologic behavior if you try to use it as scheduling > criteria. > > I'd really appreciate the views of more nova core folks on this thread, as it > looks like these blueprints have seen pretty minimal code review at this > point. H3 isn't that far away, and there is a lot of high priority things > ahead of this, and only so much coffee and review time in a day. > > -Sean > > -- > Sean Dague > http://dague.net > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
On 07/19/2013 10:37 AM, Murray, Paul (HP Cloud Services) wrote: If we agree that "something like capabilities" should go through Nova, what do you suggest should be done with the change that sparked this debate: https://review.openstack.org/#/c/35760/ I would be happy to use it or a modified version. CPU sys, user, idle, iowait time isn't capabilities though. That's a dynamically changing value. I also think the current approach where this is point in time sampling, because we only keep a single value, is going to cause some oddly pathologic behavior if you try to use it as scheduling criteria. I'd really appreciate the views of more nova core folks on this thread, as it looks like these blueprints have seen pretty minimal code review at this point. H3 isn't that far away, and there is a lot of high priority things ahead of this, and only so much coffee and review time in a day. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
If we agree that "something like capabilities" should go through Nova, what do you suggest should be done with the change that sparked this debate: https://review.openstack.org/#/c/35760/ I would be happy to use it or a modified version. Paul. -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 19 July 2013 14:28 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling On 07/19/2013 08:30 AM, Andrew Laski wrote: > On 07/19/13 at 12:08pm, Murray, Paul (HP Cloud Services) wrote: >> Hi Sean, >> >> Do you think the existing static allocators should be migrated to >> going through ceilometer - or do you see that as different? Ignoring >> backward compatibility. > > It makes sense to keep some things in Nova, in order to handle the > graceful degradation needed if Ceilometer couldn't be reached. I see > the line as something like capabilities should be handled by Nova, > memory free, vcpus available, etc... and utilization metrics handled > by Ceilometer. Yes, that makes sense to me. I'd be happy with that. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
On 07/19/2013 08:30 AM, Andrew Laski wrote: On 07/19/13 at 12:08pm, Murray, Paul (HP Cloud Services) wrote: Hi Sean, Do you think the existing static allocators should be migrated to going through ceilometer - or do you see that as different? Ignoring backward compatibility. It makes sense to keep some things in Nova, in order to handle the graceful degradation needed if Ceilometer couldn't be reached. I see the line as something like capabilities should be handled by Nova, memory free, vcpus available, etc... and utilization metrics handled by Ceilometer. Yes, that makes sense to me. I'd be happy with that. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling
On 07/19/2013 09:47 AM, Day, Phil wrote: >> -Original Message- >> From: Sean Dague [mailto:s...@dague.net] >> Sent: 19 July 2013 12:04 >> To: OpenStack Development Mailing List >> Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics >> collector for scheduling (was: New DB column or new DB table?) >> >> On 07/19/2013 06:18 AM, Day, Phil wrote: >>> Ceilometer is a great project for taking metrics available in Nova and other >> systems and making them available for use by Operations, Billing, Monitoring, >> etc - and clearly we should try and avoid having multiple collectors of the >> same >> data. >>> >>> But making the Nova scheduler dependent on Ceilometer seems to be the >> wrong way round to me - scheduling is such a fundamental operation that I >> want Nova to be self sufficient in this regard. In particular I don't want >> the >> availability of my core compute platform to be constrained by the >> availability >> of my (still evolving) monitoring system. >>> >>> If Ceilometer can be fed from the data used by the Nova scheduler then >>> that's >> a good plus - but not the other way round. >> >> I assume it would gracefully degrade to the existing static allocators if >> something went wrong. If not, well that would be very bad. >> >> Ceilometer is an integrated project in Havana. Utilization based scheduling >> would be a new feature. I'm not sure why we think that duplicating the >> metrics >> collectors in new code would be less buggy than working with Ceilometer. Nova >> depends on external projects all the time. >> >> If we have a concern about robustness here, we should be working as an >> overall >> project to address that. >> >> -Sean >> > Just to be cleat its about a lot more than just robustness in the code - its > the whole architectural pattern of putting Ceilometer at the centre of Nova > scheduling that concerns me. > > As I understand it Celiometer can collect metrics from more than one copy of > Nova - which is good; I want to run multiple independent copies in different > regions and I want to have all of my monitoring data going back to one place. > However that doesn't mean that I now also want all of those independent > copies of Nova depending on that central monitoring infrastructure for > something as basic as scheduling. (I don't want to stop anyone that does > either - but I don't see why I should be forced down that route). > > The original change that sparked this debate came not from anything to do > with utilisation based scheduling, but the pretty basic and simple desire to > add new types of consumable resource counters into the scheduler logic in a > more general way that having to make a DB schema change. This was generally > agreed to be a good thing, and it pains me to see that valuable work now > blocked on what seems to be turning into an strategic discussion around the > role of Ceilometer (Is it a monitoring tool or a fundamental metric bus, etc). > > At the point where Ceilomter can be shown to replace the current scheduler > resource mgmt code in Nova, then we should be talking about switching to it - > but in the meantime why can't we continue to have incremental improvements in > the current Nova code ? +1 > > Cheers > Phil > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?)
> -Original Message- > From: Sean Dague [mailto:s...@dague.net] > Sent: 19 July 2013 12:04 > To: OpenStack Development Mailing List > Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics > collector for scheduling (was: New DB column or new DB table?) > > On 07/19/2013 06:18 AM, Day, Phil wrote: > > Ceilometer is a great project for taking metrics available in Nova and other > systems and making them available for use by Operations, Billing, Monitoring, > etc - and clearly we should try and avoid having multiple collectors of the > same > data. > > > > But making the Nova scheduler dependent on Ceilometer seems to be the > wrong way round to me - scheduling is such a fundamental operation that I > want Nova to be self sufficient in this regard. In particular I don't want > the > availability of my core compute platform to be constrained by the availability > of my (still evolving) monitoring system. > > > > If Ceilometer can be fed from the data used by the Nova scheduler then > > that's > a good plus - but not the other way round. > > I assume it would gracefully degrade to the existing static allocators if > something went wrong. If not, well that would be very bad. > > Ceilometer is an integrated project in Havana. Utilization based scheduling > would be a new feature. I'm not sure why we think that duplicating the metrics > collectors in new code would be less buggy than working with Ceilometer. Nova > depends on external projects all the time. > > If we have a concern about robustness here, we should be working as an overall > project to address that. > > -Sean > Just to be cleat its about a lot more than just robustness in the code - its the whole architectural pattern of putting Ceilometer at the centre of Nova scheduling that concerns me. As I understand it Celiometer can collect metrics from more than one copy of Nova - which is good; I want to run multiple independent copies in different regions and I want to have all of my monitoring data going back to one place. However that doesn't mean that I now also want all of those independent copies of Nova depending on that central monitoring infrastructure for something as basic as scheduling. (I don't want to stop anyone that does either - but I don't see why I should be forced down that route). The original change that sparked this debate came not from anything to do with utilisation based scheduling, but the pretty basic and simple desire to add new types of consumable resource counters into the scheduler logic in a more general way that having to make a DB schema change. This was generally agreed to be a good thing, and it pains me to see that valuable work now blocked on what seems to be turning into an strategic discussion around the role of Ceilometer (Is it a monitoring tool or a fundamental metric bus, etc). At the point where Ceilomter can be shown to replace the current scheduler resource mgmt code in Nova, then we should be talking about switching to it - but in the meantime why can't we continue to have incremental improvements in the current Nova code ? Cheers Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?)
On 07/19/13 at 07:04am, Sean Dague wrote: On 07/19/2013 06:18 AM, Day, Phil wrote: Ceilometer is a great project for taking metrics available in Nova and other systems and making them available for use by Operations, Billing, Monitoring, etc - and clearly we should try and avoid having multiple collectors of the same data. But making the Nova scheduler dependent on Ceilometer seems to be the wrong way round to me - scheduling is such a fundamental operation that I want Nova to be self sufficient in this regard. In particular I don't want the availability of my core compute platform to be constrained by the availability of my (still evolving) monitoring system. If Ceilometer can be fed from the data used by the Nova scheduler then that's a good plus - but not the other way round. I assume it would gracefully degrade to the existing static allocators if something went wrong. If not, well that would be very bad. Ceilometer is an integrated project in Havana. Utilization based scheduling would be a new feature. I'm not sure why we think that duplicating the metrics collectors in new code would be less buggy than working with Ceilometer. Nova depends on external projects all the time. I tend to agree here. It's worth at least doing the due diligence to see if this is something that could fit into Ceilometer. Or maybe there's code that can be pulled out of Ceilometer and put into oslo to hanlde this. It doesn't make sense to duplicate effort if we can take advantage of already implemented code. If we have a concern about robustness here, we should be working as an overall project to address that. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?)
On 07/19/13 at 12:08pm, Murray, Paul (HP Cloud Services) wrote: Hi Sean, Do you think the existing static allocators should be migrated to going through ceilometer - or do you see that as different? Ignoring backward compatibility. It makes sense to keep some things in Nova, in order to handle the graceful degradation needed if Ceilometer couldn't be reached. I see the line as something like capabilities should be handled by Nova, memory free, vcpus available, etc... and utilization metrics handled by Ceilometer. The reason I ask is I want to extend the static allocators to include a couple more. These plugins are the way I would have done it. Which way do you think that should be done? Paul. -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 19 July 2013 12:04 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?) On 07/19/2013 06:18 AM, Day, Phil wrote: Ceilometer is a great project for taking metrics available in Nova and other systems and making them available for use by Operations, Billing, Monitoring, etc - and clearly we should try and avoid having multiple collectors of the same data. But making the Nova scheduler dependent on Ceilometer seems to be the wrong way round to me - scheduling is such a fundamental operation that I want Nova to be self sufficient in this regard. In particular I don't want the availability of my core compute platform to be constrained by the availability of my (still evolving) monitoring system. If Ceilometer can be fed from the data used by the Nova scheduler then that's a good plus - but not the other way round. I assume it would gracefully degrade to the existing static allocators if something went wrong. If not, well that would be very bad. Ceilometer is an integrated project in Havana. Utilization based scheduling would be a new feature. I'm not sure why we think that duplicating the metrics collectors in new code would be less buggy than working with Ceilometer. Nova depends on external projects all the time. If we have a concern about robustness here, we should be working as an overall project to address that. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?)
Hi Sean, Do you think the existing static allocators should be migrated to going through ceilometer - or do you see that as different? Ignoring backward compatibility. The reason I ask is I want to extend the static allocators to include a couple more. These plugins are the way I would have done it. Which way do you think that should be done? Paul. -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 19 July 2013 12:04 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?) On 07/19/2013 06:18 AM, Day, Phil wrote: > Ceilometer is a great project for taking metrics available in Nova and other > systems and making them available for use by Operations, Billing, Monitoring, > etc - and clearly we should try and avoid having multiple collectors of the > same data. > > But making the Nova scheduler dependent on Ceilometer seems to be the wrong > way round to me - scheduling is such a fundamental operation that I want Nova > to be self sufficient in this regard. In particular I don't want the > availability of my core compute platform to be constrained by the > availability of my (still evolving) monitoring system. > > If Ceilometer can be fed from the data used by the Nova scheduler then that's > a good plus - but not the other way round. I assume it would gracefully degrade to the existing static allocators if something went wrong. If not, well that would be very bad. Ceilometer is an integrated project in Havana. Utilization based scheduling would be a new feature. I'm not sure why we think that duplicating the metrics collectors in new code would be less buggy than working with Ceilometer. Nova depends on external projects all the time. If we have a concern about robustness here, we should be working as an overall project to address that. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling (was: New DB column or new DB table?)
On 07/19/2013 06:18 AM, Day, Phil wrote: Ceilometer is a great project for taking metrics available in Nova and other systems and making them available for use by Operations, Billing, Monitoring, etc - and clearly we should try and avoid having multiple collectors of the same data. But making the Nova scheduler dependent on Ceilometer seems to be the wrong way round to me - scheduling is such a fundamental operation that I want Nova to be self sufficient in this regard. In particular I don't want the availability of my core compute platform to be constrained by the availability of my (still evolving) monitoring system. If Ceilometer can be fed from the data used by the Nova scheduler then that's a good plus - but not the other way round. I assume it would gracefully degrade to the existing static allocators if something went wrong. If not, well that would be very bad. Ceilometer is an integrated project in Havana. Utilization based scheduling would be a new feature. I'm not sure why we think that duplicating the metrics collectors in new code would be less buggy than working with Ceilometer. Nova depends on external projects all the time. If we have a concern about robustness here, we should be working as an overall project to address that. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
