Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-24 Thread Stefan Kaltenbrunner
On 08/22/2015 06:25 AM, Tomas Vondra wrote: On 08/21/2015 08:37 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: I also don't think logging just subset of the stats is a lost case. Sure, we can't know which of the lines are important, but for example logging just the

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-22 Thread Tomas Vondra
On 08/22/2015 06:06 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: Couldn't we make it a bit smarter to handle even cases like this? For example we might first count/sum the child contexts, and then print either all child contexts (if there are only a few of them) or

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-22 Thread Tom Lane
Tomas Vondra tomas.von...@2ndquadrant.com writes: One question regarding the proposed patch though - if I get it right (just looking at the diff), it simply limits the output to first 100 child contexts at each level independently. So if each of those 100 child contexts has 100 child

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-22 Thread Tom Lane
Tomas Vondra tomas.von...@2ndquadrant.com writes: On 08/21/2015 08:37 PM, Tom Lane wrote: ... But suppose we add a parameter to memory context stats collection that is the maximum number of child contexts to print *per parent context*. If there are more than that, summarize the rest as per

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-21 Thread Tomas Vondra
On 08/21/2015 08:37 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: I also don't think logging just subset of the stats is a lost case. Sure, we can't know which of the lines are important, but for example logging just the top-level contexts with a summary of the child

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-21 Thread Tom Lane
Tomas Vondra tomas.von...@2ndquadrant.com writes: On 08/20/2015 11:04 PM, Stefan Kaltenbrunner wrote: On 08/20/2015 06:09 PM, Tom Lane wrote: (The reason I say lobotomize is that there's no particularly good reason to assume that the first N lines will tell you what you need to know. And the

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-21 Thread Tom Lane
I wrote: I thought a bit more about this. Generally, what you want to know about a given situation is which contexts have a whole lot of allocations and/or a whole lot of child contexts. What you suggest above won't work very well if the problem is buried more than about two levels down in

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Stefan Kaltenbrunner
On 08/20/2015 06:09 PM, Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I wonder if we should have a default of capping the dump to say 1k lines or such and only optionally do a full one. -1. It's worked like this for the last fifteen years or thereabouts, and you're

[HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Stefan Kaltenbrunner
Hi all! We just had a case of a very long running process of ours that creates does a lot of prepared statements through Perls DBD:Pg running into: https://rt.cpan.org/Public/Bug/Display.html?id=88827 This resulted in millions of prepared statements created, but not removed in the affected

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Tomas Vondra
Hi, On 08/20/2015 11:04 PM, Stefan Kaltenbrunner wrote: On 08/20/2015 06:09 PM, Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I wonder if we should have a default of capping the dump to say 1k lines or such and only optionally do a full one. -1. It's worked like this

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Joshua D. Drake
On 08/20/2015 08:51 AM, Stefan Kaltenbrunner wrote: This is 9.1.14 on Debian Wheezy/amd64 fwiw - but I dont think we have made relevant changes in more recent versions. It seems we may also want to consider a way to drop those prepared queries after a period time of non use. JD

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Tom Lane
Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I wonder if we should have a default of capping the dump to say 1k lines or such and only optionally do a full one. -1. It's worked like this for the last fifteen years or thereabouts, and you're the first one to complain. I suspect some