Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver

[Matt Riedemann]

On 8/2/2016 6:22 AM, John Garbutt wrote:

On 29 July 2016 at 19:58, Sean Dague  wrote:

On 07/29/2016 02:29 PM, Jay Pipes wrote:

On 07/28/2016 09:02 PM, Devananda van der Veen wrote:

On 07/28/2016 05:40 PM, Brad Morgan wrote:

I'd like to solicit some advice about potentially implementing
get_all_bw_counters() in the Ironic virt driver.

https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438
Example Implementation:
https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320


I'm ignoring the obvious question about how this data will actually be
collected/fetched as that's probably it's own topic (involving
neutron), but I
have a few questions about the Nova -> Ironic interaction:

Nova
* Is get_all_bw_counters() going to stick around for the foreseeable
future? If
not, what (if anything) is the replacement?


I don't think Nova should be in the business of monitoring *any*
transient metrics at all.

There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi,
monasca just to name a few -- that can do this work.

What action is taken if some threshold is reached is entirely
deployment-dependent and not something that Nova should care about. Nova
should just expose an API for other services to use to control the guest
instances under its management, nothing more.


More importantly... *only* xenapi driver implements this, and it's not
exposed over the API. In reality that part of the virt driver layer
should probably be removed.


AFAIK, it is only exposed via notifications:
https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/notifications/base.py#L276

I think its emitted here:
https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/compute/manager.py#L5886

Agreed with not adding to the legacy, and not to encourage new users of this.

Long term, it feels like removing this from Nova is the correct thing
to do, but I do worry about not having an obvious direct replacement
yet, and a transition plan. (This also feeds back into being able to
list deleted instances in the API, and DB soft_delete). Its not
trivial.


Like jay said, there are better tools for collecting this than Nova.


I am out of touch with what folks should use to get this data, and
build a billing system? Ceilometer + something?

It feels like that story has to be solid before we delete this
support. Maybe thats already the case, and I just don't know what that
is yet?

Thanks,
John

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



We also have the orphaned rows issue with the database which lxsli was 
trying to fix but it got messy:


https://review.openstack.org/#/q/topic:newton-db+status:abandoned

But that will break the DB archive command if you're actually using these.

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver

[John Garbutt]

On 29 July 2016 at 19:58, Sean Dague  wrote:
> On 07/29/2016 02:29 PM, Jay Pipes wrote:
>> On 07/28/2016 09:02 PM, Devananda van der Veen wrote:
>>> On 07/28/2016 05:40 PM, Brad Morgan wrote:
 I'd like to solicit some advice about potentially implementing
 get_all_bw_counters() in the Ironic virt driver.

 https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438
 Example Implementation:
 https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320


 I'm ignoring the obvious question about how this data will actually be
 collected/fetched as that's probably it's own topic (involving
 neutron), but I
 have a few questions about the Nova -> Ironic interaction:

 Nova
 * Is get_all_bw_counters() going to stick around for the foreseeable
 future? If
 not, what (if anything) is the replacement?
>>
>> I don't think Nova should be in the business of monitoring *any*
>> transient metrics at all.
>>
>> There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi,
>> monasca just to name a few -- that can do this work.
>>
>> What action is taken if some threshold is reached is entirely
>> deployment-dependent and not something that Nova should care about. Nova
>> should just expose an API for other services to use to control the guest
>> instances under its management, nothing more.
>
> More importantly... *only* xenapi driver implements this, and it's not
> exposed over the API. In reality that part of the virt driver layer
> should probably be removed.

AFAIK, it is only exposed via notifications:
https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/notifications/base.py#L276

I think its emitted here:
https://github.com/openstack/nova/blob/562a1fe9996189ddd9cc5c47ab070a498cfce258/nova/compute/manager.py#L5886

Agreed with not adding to the legacy, and not to encourage new users of this.

Long term, it feels like removing this from Nova is the correct thing
to do, but I do worry about not having an obvious direct replacement
yet, and a transition plan. (This also feeds back into being able to
list deleted instances in the API, and DB soft_delete). Its not
trivial.

> Like jay said, there are better tools for collecting this than Nova.

I am out of touch with what folks should use to get this data, and
build a billing system? Ceilometer + something?

It feels like that story has to be solid before we delete this
support. Maybe thats already the case, and I just don't know what that
is yet?

Thanks,
John

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver

[Sean Dague]

On 07/29/2016 02:29 PM, Jay Pipes wrote:
> On 07/28/2016 09:02 PM, Devananda van der Veen wrote:
>> On 07/28/2016 05:40 PM, Brad Morgan wrote:
>>> I'd like to solicit some advice about potentially implementing
>>> get_all_bw_counters() in the Ironic virt driver.
>>>
>>> https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438
>>> Example Implementation:
>>> https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320
>>>
>>>
>>> I'm ignoring the obvious question about how this data will actually be
>>> collected/fetched as that's probably it's own topic (involving
>>> neutron), but I
>>> have a few questions about the Nova -> Ironic interaction:
>>>
>>> Nova
>>> * Is get_all_bw_counters() going to stick around for the foreseeable
>>> future? If
>>> not, what (if anything) is the replacement?
> 
> I don't think Nova should be in the business of monitoring *any*
> transient metrics at all.
> 
> There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi,
> monasca just to name a few -- that can do this work.
> 
> What action is taken if some threshold is reached is entirely
> deployment-dependent and not something that Nova should care about. Nova
> should just expose an API for other services to use to control the guest
> instances under its management, nothing more.

More importantly... *only* xenapi driver implements this, and it's not
exposed over the API. In reality that part of the virt driver layer
should probably be removed.

Like jay said, there are better tools for collecting this than Nova.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver

[Jay Pipes]

On 07/28/2016 09:02 PM, Devananda van der Veen wrote:

On 07/28/2016 05:40 PM, Brad Morgan wrote:

I'd like to solicit some advice about potentially implementing
get_all_bw_counters() in the Ironic virt driver.

https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438
Example Implementation:
https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320

I'm ignoring the obvious question about how this data will actually be
collected/fetched as that's probably it's own topic (involving neutron), but I
have a few questions about the Nova -> Ironic interaction:

Nova
* Is get_all_bw_counters() going to stick around for the foreseeable future? If
not, what (if anything) is the replacement?


I don't think Nova should be in the business of monitoring *any* 
transient metrics at all.


There are many tools out there -- Nagios, collectd, HEKA, Snap, gnocchi, 
monasca just to name a few -- that can do this work.


What action is taken if some threshold is reached is entirely 
deployment-dependent and not something that Nova should care about. Nova 
should just expose an API for other services to use to control the guest 
instances under its management, nothing more.


Best,
-jay

p.s. Related: I don't think Nova should be in the business of 
implementing service group membership functionality either. We should be 
using Zookeeper or another system that was built for that purpose 
instead of continuing to maintain a wonky RDBMS-based home-grown service 
group system.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver

[Ihar Hrachyshka]

Devananda van der Veen  wrote:




On 07/28/2016 05:40 PM, Brad Morgan wrote:

I'd like to solicit some advice about potentially implementing
get_all_bw_counters() in the Ironic virt driver.

https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438
Example Implementation:
https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320

I'm ignoring the obvious question about how this data will actually be
collected/fetched as that's probably it's own topic (involving neutron),  
but I

have a few questions about the Nova -> Ironic interaction:

Nova
* Is get_all_bw_counters() going to stick around for the foreseeable  
future? If

not, what (if anything) is the replacement?

Ironic
* I assume Ironic would be responsible for knowing how to fetch bandwidth
counters for a given instance - correct?


The nova.virt.ironic driver would be responsible for implementing that  
method --
but I don't think that it makes sense to fetch that information from  
Ironic.


In some cases, it may be possible for the Node's management controller  
(eg, the

iLO) to collect/measure/expose network traffic counters for each physical
interface on the Node. None of Ironic's in-tree drivers support gathering  
this
data, afaik; Ironic isn't capturing it, and we don't have an API to  
expose it

today. If we went this route, it would be a vendor-specific thing, and not
supported by the _*ipmitool class of drivers. In other words, I don't  
think we
could have a fully open source production-oriented implementation of this  
feature.


On the other hand, with the Neutron integration now underway, if one were  
using
Neutron and OVS or OVN to manage the physical switches, then I would  
think that

Neutron could expose the bandwidth counters on the VIFs associated with the
Instance // with the user-defined Ports. I believe OVS supports this, but I
don't see anything in the Neutron API that actually exposes it...  
(IANANE, so it

may very well be there and I just didn't find it)

I'll defer to Neutron folks here. If the VIF's bandwidth counters can be  
fetched
from neutron, that would be ideal, as it should work regardless of the  
server's

management controller.

(I've added [neutron] to the subject line to solicit their input)


The only metering feature I know in neutron is for L3 metering, that  
measures traffic per router, not per port:


http://docs.openstack.org/admin-guide/networking_adv-features.html#l3-metering

It would take a completely new feature in neutron to expose traffic per  
port. I don’t think there would be a problem of backends not supporting  
this feature, but it would take some API design work.


I don’t know of any plans to expose this information through neutron API.

Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver

[Devananda van der Veen]

On 07/28/2016 05:40 PM, Brad Morgan wrote:
> I'd like to solicit some advice about potentially implementing
> get_all_bw_counters() in the Ironic virt driver.
> 
> https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438
> Example Implementation:
> https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320
> 
> I'm ignoring the obvious question about how this data will actually be
> collected/fetched as that's probably it's own topic (involving neutron), but I
> have a few questions about the Nova -> Ironic interaction:
> 
> Nova
> * Is get_all_bw_counters() going to stick around for the foreseeable future? 
> If
> not, what (if anything) is the replacement?
> 
> Ironic
> * I assume Ironic would be responsible for knowing how to fetch bandwidth
> counters for a given instance - correct?

The nova.virt.ironic driver would be responsible for implementing that method --
but I don't think that it makes sense to fetch that information from Ironic.

In some cases, it may be possible for the Node's management controller (eg, the
iLO) to collect/measure/expose network traffic counters for each physical
interface on the Node. None of Ironic's in-tree drivers support gathering this
data, afaik; Ironic isn't capturing it, and we don't have an API to expose it
today. If we went this route, it would be a vendor-specific thing, and not
supported by the _*ipmitool class of drivers. In other words, I don't think we
could have a fully open source production-oriented implementation of this 
feature.

On the other hand, with the Neutron integration now underway, if one were using
Neutron and OVS or OVN to manage the physical switches, then I would think that
Neutron could expose the bandwidth counters on the VIFs associated with the
Instance // with the user-defined Ports. I believe OVS supports this, but I
don't see anything in the Neutron API that actually exposes it... (IANANE, so it
may very well be there and I just didn't find it)

I'll defer to Neutron folks here. If the VIF's bandwidth counters can be fetched
from neutron, that would be ideal, as it should work regardless of the server's
management controller.

(I've added [neutron] to the subject line to solicit their input)


--deva

> * If so, what would this look like? (I'm assuming some Ironic API endpoint 
> Nova
> simply calls for counters - but any specific guidance here would be great.)
> 

I

> I appreciate any tips/suggestions, thank you. 
> 
> -- 
> Brad Morgan
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev