[HACKERS] Memory Context Info dump

[Rajeev rastogi]

In our customer environment as well as during development, we have observed 
that many time we need to get details of memory used by each contexts in order 
to analyze the memory consumption/leak.
So I would like to propose one contrib function interface, which will dump the 
whole memory context into a file.

One of the major infrastructure i.e. the function MemoryContextStats (which 
dump all memory context details on the console) is already there in PG but it 
is being automatically called for scenarios
such as "system is out of memory".

So Now:

1.   Create one contrib function dump_memctxt_info, which is when executed, 
it will call the existing function MemoryContextStats.

2.   The file to dump will be created in the following format:

context__.dump

3.   The file handler will be passed to MemoryContextStats.

4.   If the parameter to MemoryContextStats is NULL (All existing calls 
will pass NULL), then it will behave as earlier i.e. dump on console.

5.   The output format of the context will be same as earlier.

6.   One sample file is attached.

Use-case:
In order to check the memory leak, this contrib function can be called before 
and after execution of query. Then comparison of these two dump will be helpful 
to further analyze.

Attached is the patch. Please provide your opinion. If it is OK, I will add it 
to next commitFest.

Thanks and Regards,
Kumar Rajeev Rastogi



memory_ctxt_dumpv1.patch
Description: memory_ctxt_dumpv1.patch


context_13192_2015-09-08_181236.dump
Description: context_13192_2015-09-08_181236.dump

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Memory Context Info dump

[Greg Stark]

On Tue, Sep 8, 2015 at 4:30 PM, Tom Lane  wrote:
> As an example of potentially-more-useful aids, I'm wondering about
> tracking the high-water memory consumption of each memory context.
> (This probably wouldn't be terribly expensive if it were done at the
> granularity of malloc requests rather than individual pallocs.)
> Then perhaps something to log a context's peak usage at context
> destruction time, if it exceeds some threshold or other.


What I've been itching for in my testing is a way to save the memory
context stats, and then later print the difference between them. That
would let me see where memory was actually going to.

There are some cases where that could be tricky. If a context is
destroyed and a new one by the same name is created is it the same
context or a different one? If there are a bunch with the same name
and later there are a bunch but some are the same context and some are
new do we try to match them up, maybe sort by size?

But it would be nice to be able to see right away what context the
extra memory was allocated in between two points.



-- 
greg


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Memory Context Info dump

[Tom Lane]

Rajeev rastogi  writes:
> In our customer environment as well as during development, we have observed 
> that many time we need to get details of memory used by each contexts in 
> order to analyze the memory consumption/leak.
> So I would like to propose one contrib function interface, which will dump 
> the whole memory context into a file.

Under what circumstances would you invoke this?  Not when you were already
out of memory.

> Use-case:
> In order to check the memory leak, this contrib function can be called before 
> and after execution of query. Then comparison of these two dump will be 
> helpful to further analyze.

That's pretty uncompelling, considering that intra-query memory leaks are
one of the major concerns.  The evidence would be gone before you could
capture it.

> 1.   Create one contrib function dump_memctxt_info, which is when 
> executed, it will call the existing function MemoryContextStats.
> 2.   The file to dump will be created in the following format:
> context__.dump

Dumping to a file seems like probably the second least friendly API you
could devise ... although I'm not sure how to do better offhand.  The
existing code is made to not need to allocate any more memory, and
anything that could return data to SQL would have to do that.

I don't disagree that it would be useful to have better info available in
this area, but this patch seems like a quick hack rather than a useful
tool.

As an example of potentially-more-useful aids, I'm wondering about
tracking the high-water memory consumption of each memory context.
(This probably wouldn't be terribly expensive if it were done at the
granularity of malloc requests rather than individual pallocs.)
Then perhaps something to log a context's peak usage at context
destruction time, if it exceeds some threshold or other.

Our past discussions about tracking total usage in a context tree
would be relevant here too.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers