Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver
On 8/2/2016 6:22 AM, John Garbutt wrote: On 29 July 2016 at 19:58, Sean Dague wrote: On 07/29/2016 02:29 PM, Jay Pipes wrote: On 07/28/2016 09:02 PM, Devananda van der Veen wrote: On 07/28/2016 05:40 PM, Brad Morgan wrote: I'd like to solicit some advice about potentially implementing get_all_bw_counters() in the Ironic virt driver. https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438 Example Implementation: https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320 I'm ignoring the obvious question about how this data will actually be collected/fetched as that's probably it's own topic (involving neutron), but I have a few questions about the Nova -> Ironic interaction: Nova * Is get_all_bw_counters() going to stick around for the foreseeable future? If not, what (if anything) is the replacement? I don't think Nova should be in the business of monitoring *any* transient metrics at all. There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi, monasca just to name a few -- that can do this work. What action is taken if some threshold is reached is entirely deployment-dependent and not something that Nova should care about. Nova should just expose an API for other services to use to control the guest instances under its management, nothing more. More importantly... *only* xenapi driver implements this, and it's not exposed over the API. In reality that part of the virt driver layer should probably be removed. AFAIK, it is only exposed via notifications: https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/notifications/base.py#L276 I think its emitted here: https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/compute/manager.py#L5886 Agreed with not adding to the legacy, and not to encourage new users of this. Long term, it feels like removing this from Nova is the correct thing to do, but I do worry about not having an obvious direct replacement yet, and a transition plan. (This also feeds back into being able to list deleted instances in the API, and DB soft_delete). Its not trivial. Like jay said, there are better tools for collecting this than Nova. I am out of touch with what folks should use to get this data, and build a billing system? Ceilometer + something? It feels like that story has to be solid before we delete this support. Maybe thats already the case, and I just don't know what that is yet? Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev We also have the orphaned rows issue with the database which lxsli was trying to fix but it got messy: https://review.openstack.org/#/q/topic:newton-db+status:abandoned But that will break the DB archive command if you're actually using these. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver
On 29 July 2016 at 19:58, Sean Dague wrote: > On 07/29/2016 02:29 PM, Jay Pipes wrote: >> On 07/28/2016 09:02 PM, Devananda van der Veen wrote: >>> On 07/28/2016 05:40 PM, Brad Morgan wrote: I'd like to solicit some advice about potentially implementing get_all_bw_counters() in the Ironic virt driver. https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438 Example Implementation: https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320 I'm ignoring the obvious question about how this data will actually be collected/fetched as that's probably it's own topic (involving neutron), but I have a few questions about the Nova -> Ironic interaction: Nova * Is get_all_bw_counters() going to stick around for the foreseeable future? If not, what (if anything) is the replacement? >> >> I don't think Nova should be in the business of monitoring *any* >> transient metrics at all. >> >> There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi, >> monasca just to name a few -- that can do this work. >> >> What action is taken if some threshold is reached is entirely >> deployment-dependent and not something that Nova should care about. Nova >> should just expose an API for other services to use to control the guest >> instances under its management, nothing more. > > More importantly... *only* xenapi driver implements this, and it's not > exposed over the API. In reality that part of the virt driver layer > should probably be removed. AFAIK, it is only exposed via notifications: https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/notifications/base.py#L276 I think its emitted here: https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/compute/manager.py#L5886 Agreed with not adding to the legacy, and not to encourage new users of this. Long term, it feels like removing this from Nova is the correct thing to do, but I do worry about not having an obvious direct replacement yet, and a transition plan. (This also feeds back into being able to list deleted instances in the API, and DB soft_delete). Its not trivial. > Like jay said, there are better tools for collecting this than Nova. I am out of touch with what folks should use to get this data, and build a billing system? Ceilometer + something? It feels like that story has to be solid before we delete this support. Maybe thats already the case, and I just don't know what that is yet? Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver
On 07/29/2016 02:29 PM, Jay Pipes wrote: > On 07/28/2016 09:02 PM, Devananda van der Veen wrote: >> On 07/28/2016 05:40 PM, Brad Morgan wrote: >>> I'd like to solicit some advice about potentially implementing >>> get_all_bw_counters() in the Ironic virt driver. >>> >>> https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438 >>> Example Implementation: >>> https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320 >>> >>> >>> I'm ignoring the obvious question about how this data will actually be >>> collected/fetched as that's probably it's own topic (involving >>> neutron), but I >>> have a few questions about the Nova -> Ironic interaction: >>> >>> Nova >>> * Is get_all_bw_counters() going to stick around for the foreseeable >>> future? If >>> not, what (if anything) is the replacement? > > I don't think Nova should be in the business of monitoring *any* > transient metrics at all. > > There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi, > monasca just to name a few -- that can do this work. > > What action is taken if some threshold is reached is entirely > deployment-dependent and not something that Nova should care about. Nova > should just expose an API for other services to use to control the guest > instances under its management, nothing more. More importantly... *only* xenapi driver implements this, and it's not exposed over the API. In reality that part of the virt driver layer should probably be removed. Like jay said, there are better tools for collecting this than Nova. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver
On 07/28/2016 09:02 PM, Devananda van der Veen wrote: On 07/28/2016 05:40 PM, Brad Morgan wrote: I'd like to solicit some advice about potentially implementing get_all_bw_counters() in the Ironic virt driver. https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438 Example Implementation: https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320 I'm ignoring the obvious question about how this data will actually be collected/fetched as that's probably it's own topic (involving neutron), but I have a few questions about the Nova -> Ironic interaction: Nova * Is get_all_bw_counters() going to stick around for the foreseeable future? If not, what (if anything) is the replacement? I don't think Nova should be in the business of monitoring *any* transient metrics at all. There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi, monasca just to name a few -- that can do this work. What action is taken if some threshold is reached is entirely deployment-dependent and not something that Nova should care about. Nova should just expose an API for other services to use to control the guest instances under its management, nothing more. Best, -jay p.s. Related: I don't think Nova should be in the business of implementing service group membership functionality either. We should be using Zookeeper or another system that was built for that purpose instead of continuing to maintain a wonky RDBMS-based home-grown service group system. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver
Devananda van der Veen wrote: On 07/28/2016 05:40 PM, Brad Morgan wrote: I'd like to solicit some advice about potentially implementing get_all_bw_counters() in the Ironic virt driver. https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438 Example Implementation: https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320 I'm ignoring the obvious question about how this data will actually be collected/fetched as that's probably it's own topic (involving neutron), but I have a few questions about the Nova -> Ironic interaction: Nova * Is get_all_bw_counters() going to stick around for the foreseeable future? If not, what (if anything) is the replacement? Ironic * I assume Ironic would be responsible for knowing how to fetch bandwidth counters for a given instance - correct? The nova.virt.ironic driver would be responsible for implementing that method -- but I don't think that it makes sense to fetch that information from Ironic. In some cases, it may be possible for the Node's management controller (eg, the iLO) to collect/measure/expose network traffic counters for each physical interface on the Node. None of Ironic's in-tree drivers support gathering this data, afaik; Ironic isn't capturing it, and we don't have an API to expose it today. If we went this route, it would be a vendor-specific thing, and not supported by the _*ipmitool class of drivers. In other words, I don't think we could have a fully open source production-oriented implementation of this feature. On the other hand, with the Neutron integration now underway, if one were using Neutron and OVS or OVN to manage the physical switches, then I would think that Neutron could expose the bandwidth counters on the VIFs associated with the Instance // with the user-defined Ports. I believe OVS supports this, but I don't see anything in the Neutron API that actually exposes it... (IANANE, so it may very well be there and I just didn't find it) I'll defer to Neutron folks here. If the VIF's bandwidth counters can be fetched from neutron, that would be ideal, as it should work regardless of the server's management controller. (I've added [neutron] to the subject line to solicit their input) The only metering feature I know in neutron is for L3 metering, that measures traffic per router, not per port: http://docs.openstack.org/admin-guide/networking_adv-features.html#l3-metering It would take a completely new feature in neutron to expose traffic per port. I don’t think there would be a problem of backends not supporting this feature, but it would take some API design work. I don’t know of any plans to expose this information through neutron API. Ihar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver
On 07/28/2016 05:40 PM, Brad Morgan wrote: > I'd like to solicit some advice about potentially implementing > get_all_bw_counters() in the Ironic virt driver. > > https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438 > Example Implementation: > https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320 > > I'm ignoring the obvious question about how this data will actually be > collected/fetched as that's probably it's own topic (involving neutron), but I > have a few questions about the Nova -> Ironic interaction: > > Nova > * Is get_all_bw_counters() going to stick around for the foreseeable future? > If > not, what (if anything) is the replacement? > > Ironic > * I assume Ironic would be responsible for knowing how to fetch bandwidth > counters for a given instance - correct? The nova.virt.ironic driver would be responsible for implementing that method -- but I don't think that it makes sense to fetch that information from Ironic. In some cases, it may be possible for the Node's management controller (eg, the iLO) to collect/measure/expose network traffic counters for each physical interface on the Node. None of Ironic's in-tree drivers support gathering this data, afaik; Ironic isn't capturing it, and we don't have an API to expose it today. If we went this route, it would be a vendor-specific thing, and not supported by the _*ipmitool class of drivers. In other words, I don't think we could have a fully open source production-oriented implementation of this feature. On the other hand, with the Neutron integration now underway, if one were using Neutron and OVS or OVN to manage the physical switches, then I would think that Neutron could expose the bandwidth counters on the VIFs associated with the Instance // with the user-defined Ports. I believe OVS supports this, but I don't see anything in the Neutron API that actually exposes it... (IANANE, so it may very well be there and I just didn't find it) I'll defer to Neutron folks here. If the VIF's bandwidth counters can be fetched from neutron, that would be ideal, as it should work regardless of the server's management controller. (I've added [neutron] to the subject line to solicit their input) --deva > * If so, what would this look like? (I'm assuming some Ironic API endpoint > Nova > simply calls for counters - but any specific guidance here would be great.) > I > I appreciate any tips/suggestions, thank you. > > -- > Brad Morgan > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev