Re: [openstack-dev] [requirements] Adding objgraph to global requirements

[Doug Hellmann]

On Mar 30, 2018, at 4:08 AM, Renat Akhmerov > wrote:

> On 29 Mar 2018, 20:20 +0700, Doug Hellmann  >, wrote:
> 
>> What sorts of tools are you talking about building? 
> 
> Well, some more high level functions to diagnose memory usage, especially 
> aids to help find memory leaks. Or not even necessary memory leaks, we may 
> also want/need to reduce the memory footprint. Although objgraph is very 
> helpful on its own it is still pretty low level and, for example, if we want 
> to inject some diagnostic message somewhere in the code (e.g. “top 5 object 
> types whose total size increased memory consumption since the last call”) 
> we’d have to directly invoke objgraph and remember what exact parameters to 
> use to achieve that. I think it’d be more convenient to have some more 
> high-level functions that already encapsulate all necessary logic related to 
> using objgraph and achieve what’s needed. Also, I wouldn’t want to spread out 
> uses of objgraph all of the code, I would better have one module that uses 
> it. Just to reduce coupling with a 3rd party lib.
> 
> Sorry if that sounds too abstract now, I’ll try to come with concrete 
> examples to demonstrate what I mean.
> 
> Renat
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org 
> ?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 

It sounds like you have ideas for things that could either live in objgraph 
directly or a new library that wraps objgraph. Either way I think it’s a good 
idea to build something concrete so we can see the impact of integrating it. 

You might want to have a look at oslo.reports as an integration point. 

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] Adding objgraph to global requirements

[Renat Akhmerov]

On 29 Mar 2018, 20:20 +0700, Doug Hellmann , wrote:

> What sorts of tools are you talking about building?

Well, some more high level functions to diagnose memory usage, especially aids 
to help find memory leaks. Or not even necessary memory leaks, we may also 
want/need to reduce the memory footprint. Although objgraph is very helpful on 
its own it is still pretty low level and, for example, if we want to inject 
some diagnostic message somewhere in the code (e.g. “top 5 object types whose 
total size increased memory consumption since the last call”) we’d have to 
directly invoke objgraph and remember what exact parameters to use to achieve 
that. I think it’d be more convenient to have some more high-level functions 
that already encapsulate all necessary logic related to using objgraph and 
achieve what’s needed. Also, I wouldn’t want to spread out uses of objgraph all 
of the code, I would better have one module that uses it. Just to reduce 
coupling with a 3rd party lib.

Sorry if that sounds too abstract now, I’ll try to come with concrete examples 
to demonstrate what I mean.

Renat
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] Adding objgraph to global requirements

[Doug Hellmann]

Excerpts from Renat Akhmerov's message of 2018-03-29 15:33:58 +0700:
> After some discussion in IRC on this topic there was an idea just to write 
> and push upstream needed tools using objgraph w/o having it in the 
> requirements.txt at all. We just need to make sure that those tools are never 
> used during production runs and unit tests (CI will help to verify that). If 
> needed, objgraph can be manually installed used when we need to investigate 
> something.
> 
> If such practice is considered OK and doesn’t violate any OpenStack 
> guidelines then I think this would work, at least in my case.

I don't see any problem with that but I'm also not necessarily opposed
to adding it to the global requirements list so we can use it like other
dependencies.

What sorts of tools are you talking about building?

> 
> Thanks
> 
> Renat Akhmerov
> @Nokia
> 
> On 29 Mar 2018, 15:00 +0700, Renat Akhmerov , wrote:
> > Hi,
> >
> > Can we consider to add objgraph [1] to OpenStack global requirements? I 
> > found this library extremely useful for investigating memory leaks in 
> > Python programs but unfortunately I can’t push upstream any code using it. 
> > It seems to be pretty mature and supports all needed Python versions.
> >
> > Or maybe there’s some alternative already available in the OpenStack 
> > requirements?
> >
> > [1] https://pypi.python.org/pypi/objgraph/3.4.0
> >
> >
> > Thanks
> >
> > Renat Akhmerov
> > @Nokia

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] Adding objgraph to global requirements

[Renat Akhmerov]

After some discussion in IRC on this topic there was an idea just to write and 
push upstream needed tools using objgraph w/o having it in the requirements.txt 
at all. We just need to make sure that those tools are never used during 
production runs and unit tests (CI will help to verify that). If needed, 
objgraph can be manually installed used when we need to investigate something.

If such practice is considered OK and doesn’t violate any OpenStack guidelines 
then I think this would work, at least in my case.

Thanks

Renat Akhmerov
@Nokia

On 29 Mar 2018, 15:00 +0700, Renat Akhmerov , wrote:
> Hi,
>
> Can we consider to add objgraph [1] to OpenStack global requirements? I found 
> this library extremely useful for investigating memory leaks in Python 
> programs but unfortunately I can’t push upstream any code using it. It seems 
> to be pretty mature and supports all needed Python versions.
>
> Or maybe there’s some alternative already available in the OpenStack 
> requirements?
>
> [1] https://pypi.python.org/pypi/objgraph/3.4.0
>
>
> Thanks
>
> Renat Akhmerov
> @Nokia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [requirements] Adding objgraph to global requirements

[Renat Akhmerov]

Hi,

Can we consider to add objgraph [1] to OpenStack global requirements? I found 
this library extremely useful for investigating memory leaks in Python programs 
but unfortunately I can’t push upstream any code using it. It seems to be 
pretty mature and supports all needed Python versions.

Or maybe there’s some alternative already available in the OpenStack 
requirements?

[1] https://pypi.python.org/pypi/objgraph/3.4.0


Thanks

Renat Akhmerov
@Nokia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev