Re: [HACKERS] Memory leaks in record_out and record_send
On Tue, Nov 13, 2012 at 05:50:08PM -0500, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > Yeah. The thing that concerns me is that I think we have a pretty > > decent number of memory contexts where the expected number of > > allocations is very small ... and we have the context *just in case* > > we do more than that in certain instances. I've seen profiles where > > the setup/teardown costs of memory contexts are significant ... which > > doesn't mean that those examples would perform better with fewer > > memory contexts, but it's enough to make me pause for thought. > > So, for my 2c, I'm on the other side of this, personally. We have > memory contexts for more-or-less exactly this issue. It's one of the > great things about PG- it's resiliant and very unlikely to have large or > bad memory leaks in general, much of which can, imv, be attributed to > our use of memory contexts. If the problem is that we create memory context overhead which is not necessary in many cases, perhaps we can reduce the overhead somehow. IIRC we have a seperate function for resetting a context and freeing it entirely. If there was a quick test we could do such that resetting a context did nothing unless at least (say) 16k had been allocated, that might reduce the cost for many very small allocations. Ofcourse, unless someone comes up with a way to measure the cost this is all handwaving, but it might a nice project for someone interested in learning to hack postgres. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > He who writes carelessly confesses thereby at the very outset that he does > not attach much importance to his own thoughts. -- Arthur Schopenhauer signature.asc Description: Digital signature
Re: [HACKERS] Memory leaks in record_out and record_send
* Robert Haas (robertmh...@gmail.com) wrote: > Yeah. The thing that concerns me is that I think we have a pretty > decent number of memory contexts where the expected number of > allocations is very small ... and we have the context *just in case* > we do more than that in certain instances. I've seen profiles where > the setup/teardown costs of memory contexts are significant ... which > doesn't mean that those examples would perform better with fewer > memory contexts, but it's enough to make me pause for thought. So, for my 2c, I'm on the other side of this, personally. We have memory contexts for more-or-less exactly this issue. It's one of the great things about PG- it's resiliant and very unlikely to have large or bad memory leaks in general, much of which can, imv, be attributed to our use of memory contexts. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Memory leaks in record_out and record_send
On Tue, Nov 13, 2012 at 3:21 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Nov 13, 2012 at 12:18 AM, Tom Lane wrote: >>> I wonder though if we ought to think about running output functions in >>> a short-lived memory context instead of the executor's main context. >>> We've considered that before, I think, and it's always been the path >>> of least resistance to fix the output functions instead --- but there >>> will always be another leak I'm afraid. > >> Such is the lot of people who code in C. I worry that the number of >> memory contexts we're kicking around already is imposing a significant >> distributed overhead on the system that is hard to measure but >> nevertheless real, and that this will add to it. > > Yeah, perhaps. I'd like to think that a MemoryContextReset is cheaper > than a bunch of retail pfree's, but it's hard to prove anything without > actually coding and testing it --- and on modern machines, effects like > cache locality could swamp pure instruction-count gains anyway. Yeah. The thing that concerns me is that I think we have a pretty decent number of memory contexts where the expected number of allocations is very small ... and we have the context *just in case* we do more than that in certain instances. I've seen profiles where the setup/teardown costs of memory contexts are significant ... which doesn't mean that those examples would perform better with fewer memory contexts, but it's enough to make me pause for thought. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leaks in record_out and record_send
Robert Haas writes: > On Tue, Nov 13, 2012 at 12:18 AM, Tom Lane wrote: >> I wonder though if we ought to think about running output functions in >> a short-lived memory context instead of the executor's main context. >> We've considered that before, I think, and it's always been the path >> of least resistance to fix the output functions instead --- but there >> will always be another leak I'm afraid. > Such is the lot of people who code in C. I worry that the number of > memory contexts we're kicking around already is imposing a significant > distributed overhead on the system that is hard to measure but > nevertheless real, and that this will add to it. Yeah, perhaps. I'd like to think that a MemoryContextReset is cheaper than a bunch of retail pfree's, but it's hard to prove anything without actually coding and testing it --- and on modern machines, effects like cache locality could swamp pure instruction-count gains anyway. Anyway, I committed the narrow fix for the moment. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leaks in record_out and record_send
On Tue, Nov 13, 2012 at 12:18 AM, Tom Lane wrote: > I wonder though if we ought to think about running output functions in > a short-lived memory context instead of the executor's main context. > We've considered that before, I think, and it's always been the path > of least resistance to fix the output functions instead --- but there > will always be another leak I'm afraid. Such is the lot of people who code in C. I worry that the number of memory contexts we're kicking around already is imposing a significant distributed overhead on the system that is hard to measure but nevertheless real, and that this will add to it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leaks in record_out and record_send
Dimitri Fontaine writes: > Tom Lane writes: >> I think explicit calls like that actually wouldn't be a problem, >> since they'd be run in a per-tuple context anyway. The cases that >> are problematic are hard-coded I/O function calls. I'm worried >> about the ones like, say, plpgsql's built-in conversion operations. >> We could probably fix printtup's usage with some confidence, but >> there are a lot of other ones. > That's a good reason to get them into a shorter memory context, but > which? per transaction maybe? shorter? It would have to be per-tuple to do any good. The existing behavior is per-query and causes problems if lots of rows are output. In plpgsql it would be a function-call-lifespan leak. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leaks in record_out and record_send
Tom Lane writes: > I think explicit calls like that actually wouldn't be a problem, > since they'd be run in a per-tuple context anyway. The cases that > are problematic are hard-coded I/O function calls. I'm worried > about the ones like, say, plpgsql's built-in conversion operations. > We could probably fix printtup's usage with some confidence, but > there are a lot of other ones. That's a good reason to get them into a shorter memory context, but which? per transaction maybe? shorter? -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leaks in record_out and record_send
Dimitri Fontaine writes: > Tom Lane writes: >> Thoughts? > I've been using textin(record_out(NEW)) in generic partitioning > triggers, and you can find examples of that trick in the wiki, so I > think we have users of that in the field. I think explicit calls like that actually wouldn't be a problem, since they'd be run in a per-tuple context anyway. The cases that are problematic are hard-coded I/O function calls. I'm worried about the ones like, say, plpgsql's built-in conversion operations. We could probably fix printtup's usage with some confidence, but there are a lot of other ones. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leaks in record_out and record_send
Tom Lane writes: > OTOH I can't see trying to back-patch a solution like that. If we want > to fix this in the back branches (and note the complaint linked above is > against 8.3), I think we have to do it as attached. > > Thoughts? I've been using textin(record_out(NEW)) in generic partitioning triggers, and you can find examples of that trick in the wiki, so I think we have users of that in the field. Please indeed do consider backpatching! I don't have an opinion on the opportunity to use a shorter memory context, I feel that would need some more involved analytics than my brainpower of the moment allows me to consider. Thanks, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leaks on SRF rescan
Neil Conway <[EMAIL PROTECTED]> writes: > On Thu, 2008-02-21 at 21:42 -0500, Tom Lane wrote: >> Yeah. I think it's hopeless to expect these functions to all hew to >> the straight and narrow path. It seems to me that the right way is for >> the sub-select to somehow run in its own "per-query" context. > Hmm, I was thinking of just fixing this by arranging for the > FuncCallContext's multi-call context to be a special context created by > the function scan, and that is reset/deleted at the appropriate time. > Would this not fix the issue as well? That might work, particularly if we could arrange for all the functions invoked in a particular subquery to share the same "per query" context. Want to take a whack at it? regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Memory leaks on SRF rescan
On Thu, 2008-02-21 at 21:42 -0500, Tom Lane wrote: > Given your point (2), is this worth fixing by itself? Right, probably not. > Yeah. I think it's hopeless to expect these functions to all hew to > the straight and narrow path. It seems to me that the right way is for > the sub-select to somehow run in its own "per-query" context. Hmm, I was thinking of just fixing this by arranging for the FuncCallContext's multi-call context to be a special context created by the function scan, and that is reset/deleted at the appropriate time. Would this not fix the issue as well? -Neil ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Memory leaks on SRF rescan
Neil Conway <[EMAIL PROTECTED]> writes: > Two different classes of allocations in the EState's per-query context > are leaked: > (1) In FunctionNext(), we call ExecMakeTableFunctionResult() to compute > the result set of the SRF, which allocates the TupleDesc out-parameter > in the per-query memory context. Since rescan of a function scan (with > chgParam != NULL) results in taking this code path again, we can leak 1 > TupleDesc for each rescan of a function scan. I think this is plainly a > bug -- the first attached patch fixes it. Given your point (2), is this worth fixing by itself? > (2) In various SRFs, allocations are made in the "multi-call context" > but are not released before calling SRF_RETURN_DONE. Yeah. I think it's hopeless to expect these functions to all hew to the straight and narrow path. It seems to me that the right way is for the sub-select to somehow run in its own "per-query" context. Not sure about the implications of that bit of arm-waving ... it might be a bit tricky since I think we expect the executor state tree to get set up only once. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Memory leaks
Greg Copeland wrote: I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Is this done periodically? If so, what tools are others using? I'm currently using dmalloc for my curiosity. [1] Not sure yet as I'm really wanting to find culumative leaks rather than one shot allocations which are simply never freed prior to process termination. While all leaks should be fixed, obviously, this is one of the "good" things in the parennial "Thread vs process" argument for processes. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
Greg Copeland <[EMAIL PROTECTED]> writes: > Ya, I'm currently looking to see how the memory is being used and why. > I'm trying to better understand it's life cycle. You implying that even > the short term memory should be using the palloc stuff? What about long > term? Blanket statement that pretty much all the PLy stuff should > really be using palloc? Short-term stuff almost certainly should be using palloc, IMHO; anything that is not going to survive the current function invocation should be palloc'd in CurrentMemoryContext. The main reason for this is that you don't need to fear leaking such memory if the function is aborted by elog(). Depending on what you are doing, you may not have to bother with explicit pfree's at all for such stuff. (In a PL handler you could probably get away with omitting pfree's for stuff allocated once per call, but not for stuff allocated once per statement. Unless you were to make a new context that gets reset for each statement ... which might be a good idea.) For stuff that is going to live a long time and then be explicitly freed, I don't think there's a hard-and-fast rule about which to use. If you are building a complex data structure (parsetree, say) then it's going to be easier to build it in a memory context and then just free the context rather than tearing down the data structure piece-by-piece. But when you are talking about a single object, there's not a heck of a lot of difference between malloc() and palloc in TopMemoryContext. I'd lean towards using the palloc routines anyway, for consistency of coding style, but that's a judgment call not a must-do thing. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Memory leaks
On Wed, 2002-10-23 at 08:48, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > Okay. I've started looking at plpython to better understand it's memory > > needs. I'm seeing a mix of mallocs and PLy_malloc. The PLy version is > > basically malloc which also checks and reports on memory allocation > > errors. Anyone know if the cases where malloc was used was purposely > > done so for performance reasons or simply the flavor or the day? > > Probably either oversight or the result of different people's different > coding styles. My local copy has this changed to PLy stuff now. Testing shows it's good...then again, I didn't really expect it to change anything. I'll submit patches later. > > > I thinking for starters, the plpython module could be normalized to use > > the PLy_malloc stuff across the board. Then again, I still need to > > spend some more time on it. ;) > > Consistency is good. What I'd wonder about, though, is whether you > shouldn't be using palloc ;-). malloc, with or without a PLy_ wrapper, > doesn't provide any leverage to help you get rid of stuff when you don't > want it anymore. Ya, I'm currently looking to see how the memory is being used and why. I'm trying to better understand it's life cycle. You implying that even the short term memory should be using the palloc stuff? What about long term? Blanket statement that pretty much all the PLy stuff should really be using palloc? > > >>> Well, the thing that really got my attention is that dmalloc is > >>> reporting frees on null pointers. > >> > >> AFAIK that would dump core on many platforms (it sure does here...), > > I have to take that back: I was thinking about pfree() not free(). > The ANSI C spec says that free(NULL) is a legal no-op, and there are Oh really. I didn't realize that. I've been using the "if( ptr ) " stuff for so long I didn't realize I didn't need to anymore. Thanks for the update. That was, of course, the cause for alarm. > It's > probably pointless to try to convince people to change that coding style. Well at this late time, I think it's safe to say that it's not causing problems for anyone on any of the supported platforms. So I'll not waste time looking for it even though I happen think it's a poor practice just the same. Thanks, Greg ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Memory leaks
Greg Copeland <[EMAIL PROTECTED]> writes: > Okay. I've started looking at plpython to better understand it's memory > needs. I'm seeing a mix of mallocs and PLy_malloc. The PLy version is > basically malloc which also checks and reports on memory allocation > errors. Anyone know if the cases where malloc was used was purposely > done so for performance reasons or simply the flavor or the day? Probably either oversight or the result of different people's different coding styles. > I thinking for starters, the plpython module could be normalized to use > the PLy_malloc stuff across the board. Then again, I still need to > spend some more time on it. ;) Consistency is good. What I'd wonder about, though, is whether you shouldn't be using palloc ;-). malloc, with or without a PLy_ wrapper, doesn't provide any leverage to help you get rid of stuff when you don't want it anymore. >>> Well, the thing that really got my attention is that dmalloc is >>> reporting frees on null pointers. >> >> AFAIK that would dump core on many platforms (it sure does here...), I have to take that back: I was thinking about pfree() not free(). The ANSI C spec says that free(NULL) is a legal no-op, and there are just a few ancient C libraries (perhaps none anymore) where it'll crash. I tend to do "if (ptr) free(ptr)" from force of habit, but I notice that psql (among other places) relies heavily on the ANSI behavior. It's probably pointless to try to convince people to change that coding style. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Memory leaks
"Nigel J. Andrews" <[EMAIL PROTECTED]> writes: > On Tue, 22 Oct 2002, Tom Lane wrote: >> Not everywhere. plpgsql is full of malloc's and I think the other PL >> modules are too --- and that's not to mention the allocation policies of >> the perl, tcl, etc, language interpreters... > I was going to make the suggestion that malloc et al. could be replaced with > palloc etc but then that raises too many complications without just shooving > everything into a long lived context anyway. Also I think we've got to rely on, > i.e. it is sensible to do so, the underlying language handling memory > correctly. If the perl/tcl/etc interpreters have internal memory leaks, there's little we can do about that except file bug reports. What I was wondering about is whether there isn't deliberate action we need to take to inform those interpreters when data is no longer required. An example: when a procedure is updated with CREATE OR REPLACE FUNCTION, the only thing pltcl does about it is a solitary Tcl_DeleteHashEntry(); it doesn't try to tell Tcl to delete the existing Tcl procedure. That might happen for free (will we always regenerate the same Tcl procedure name? not sure), but if the omission causes a leak it's surely not Tcl's fault. That's on top of our own data structures about the pltcl function (pltcl_proc_desc and related stuff), which are definitely leaked in this scenario. Sticking all the data about a given function into a memory context that's specific to the function would make it easier to reclaim our own memory in this scenario, but we'd still have to tell Tcl to clean up its own memory. Actually pltcl's internal structures about a function look simple enough that it may not be worth using a context for them. It would definitely be useful to do that for plpgsql, though, which builds an extremely complicated structure for each function (and leaks it all on function redefinition :-(). regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Memory leaks
On Tue, 2002-10-22 at 22:28, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > >So again, I'm not really sure it they are meaningful at > > this point. > > psql might well have some internal leaks; the backend memory-context > design doesn't apply to it. Okay. Thanks. I'll probably take another look at it a little later and report back if I find anything of value. > >Does that mean they are not using > > palloc/pfree stuff? > > Not everywhere. plpgsql is full of malloc's and I think the other PL > modules are too --- and that's not to mention the allocation policies of > the perl, tcl, etc, language interpreters. We could use a thorough > review of that whole area. > Okay. I've started looking at plpython to better understand it's memory needs. I'm seeing a mix of mallocs and PLy_malloc. The PLy version is basically malloc which also checks and reports on memory allocation errors. Anyone know if the cases where malloc was used was purposely done so for performance reasons or simply the flavor or the day? I thinking for starters, the plpython module could be normalized to use the PLy_malloc stuff across the board. Then again, I still need to spend some more time on it. ;) > > Well, the thing that really got my attention is that dmalloc is > > reporting frees on null pointers. > > AFAIK that would dump core on many platforms (it sure does here...), > so I don't think I believe it without seeing chapter and verse. I actually expected it to do that here on my x86-Linux platform but a quick check showed that it was quiet happy with it. What platforms are you using -- just curious? > But if you can point out where it's really happening, then we must fix it. > I'll trying track this down some more this coming week to see if this is really occurring. After thinking about it, I'm not sure why dmalloc would ever report a free on null if it were not actually occurring. After all, any call to free should still be showing some memory address (valid or otherwise). Off the top of my head, I can't think of an artifact that would cause it to falsely report it. Greg ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On Tue, 22 Oct 2002, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > > Interesting. Having not looked at memory management schemes used in the > > pl implementations, can you enlighten me by what you mean by "integrate > > the memory-context notion"? Does that mean they are not using > > palloc/pfree stuff? > > Not everywhere. plpgsql is full of malloc's and I think the other PL > modules are too --- and that's not to mention the allocation policies of > the perl, tcl, etc, language interpreters... I was going to make the suggestion that malloc et al. could be replaced with palloc etc but then that raises too many complications without just shooving everything into a long lived context anyway. Also I think we've got to rely on, i.e. it is sensible to do so, the underlying language handling memory correctly. Hmmm...there do seem to be a few mallocs in plpython.c . I haven't looked very closely but nothing jumped out at me as being obviously wrong from the grep output. -- Nigel J. Andrews ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On Tue, Oct 22, 2002 at 11:28:23PM -0400, Tom Lane wrote: > > I then moved on to psql, again, just for fun. Here, I'm thinking that I > > started to find some other leaks...but again, I've not spent any real > > time on it. So again, I'm not really sure it they are meaningful at > > this point. > > psql might well have some internal leaks; the backend memory-context > design doesn't apply to it. But why? In the Mape project is used mmgr based on PostgreSQL's mmgr and it's used for BE and FE. There is not problem with it (BTW backend is multithread:-). IMHO use memory-context design for FE is good idea if FE a lot works with memory. I already long time think about shared lib with PostgreSQL mmgr... Karel -- Karel Zak <[EMAIL PROTECTED]> http://home.zf.jcu.cz/~zakkr/ C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Memory leaks
Greg Copeland <[EMAIL PROTECTED]> writes: > On Tue, 2002-10-22 at 17:09, Tom Lane wrote: >> Yes, this has been dealt with before. > What tools, aside from noggin v1.0, did they use? Do we know? s/they/me/ ... none. But I don't know of any that I think would be useful. > I then moved on to psql, again, just for fun. Here, I'm thinking that I > started to find some other leaks...but again, I've not spent any real > time on it. So again, I'm not really sure it they are meaningful at > this point. psql might well have some internal leaks; the backend memory-context design doesn't apply to it. >> Possibly the best answer is to integrate the memory-context notion into >> those modules; if they did most of their work in a temp context that >> could be freed once per PL statement or so, the problems would pretty >> much go away. > Interesting. Having not looked at memory management schemes used in the > pl implementations, can you enlighten me by what you mean by "integrate > the memory-context notion"? Does that mean they are not using > palloc/pfree stuff? Not everywhere. plpgsql is full of malloc's and I think the other PL modules are too --- and that's not to mention the allocation policies of the perl, tcl, etc, language interpreters. We could use a thorough review of that whole area. > Well, the thing that really got my attention is that dmalloc is > reporting frees on null pointers. AFAIK that would dump core on many platforms (it sure does here...), so I don't think I believe it without seeing chapter and verse. But if you can point out where it's really happening, then we must fix it. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Memory leaks
"Nigel J. Andrews" <[EMAIL PROTECTED]> writes: > I saw use of a couple of malloc (or Python specific malloc) calls the > other day but I also seem to recall that, after consideration, I > decided the memory needed to survive for the duration of the > backend. Should I have created a new child of the top context and > changed these malloc calls? If there is truly no reason to release the memory before the backend dies, then I see no reason to avoid malloc(). Basically what the memory context stuff gives you is the ability to bunch allocations together and release a whole tree of stuff for little work. An example: currently, the plpgsql parser builds its parsed syntax tree with a bunch of malloc's. It has no way to free a syntax tree, so that tree lives till the backend exits, whether it's ever used again or not. It would be better to build the tree via palloc's in a context created specially for the specific function: then we could free the whole context if we discovered that the function had gone away or been redefined. (Compare what relcache does for rule syntaxtrees for rules associated with relcache entries.) But right at the moment, there's no mechanism to discover that situation, and so there's not now any direct value in using palloc instead of malloc for plpgsql syntaxtrees. Yet that's surely the direction to go in for the long term. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On 22 Oct 2002, Greg Copeland wrote: > On Tue, 2002-10-22 at 17:09, Tom Lane wrote: > > > plpgsql has some issues too, I suspect, but not as bad as pltcl etc. > > Possibly the best answer is to integrate the memory-context notion into > > those modules; if they did most of their work in a temp context that > > could be freed once per PL statement or so, the problems would pretty > > much go away. > > Interesting. Having not looked at memory management schemes used in the > pl implementations, can you enlighten me by what you mean by "integrate > the memory-context notion"? Does that mean they are not using > palloc/pfree stuff? I saw use of a couple of malloc (or Python specific malloc) calls the other day but I also seem to recall that, after consideration, I decided the memory needed to survive for the duration of the backend. Should I have created a new child of the top context and changed these malloc calls? I was going to ask about thoughts on redirecting malloc etc to palloc etc and thereby intercepting memory allocation within the languages and automatically bringing them into the memory context realm. However, that would just be making life way too awkward, bearing in mind the above paragraph. Can't we get Sir Mongle (or whatever the name was) to test these things under the auspices of them being DoS attacks? -- Nigel J. Andrews ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Memory leaks
On Tue, 2002-10-22 at 17:09, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > I've started playing a little with Postgres to determine if there were > > memory leaks running around. After some very brief checking, I'm > > starting[1] to think that the answer is yes. Has anyone already gone > > through a significant effort to locate and eradicate memory leaks? > > Yes, this has been dealt with before. What tools, aside from noggin v1.0, did they use? Do we know? > Have you read > src/backend/utils/mmgr/README? Yes...but it's been some time since I last opened it. It was because I knew there were some caveats that I wasn't attempting to claim for sure that there were leaks. I then moved on to psql, again, just for fun. Here, I'm thinking that I started to find some other leaks...but again, I've not spent any real time on it. So again, I'm not really sure it they are meaningful at this point. > > AFAIK the major problems these days are not in the backend as a whole, > but in the lesser-used PL language modules (cf recent discussions). Ya, that's what made me wonder about the topic on a larger scale. > plpgsql has some issues too, I suspect, but not as bad as pltcl etc. > Possibly the best answer is to integrate the memory-context notion into > those modules; if they did most of their work in a temp context that > could be freed once per PL statement or so, the problems would pretty > much go away. Interesting. Having not looked at memory management schemes used in the pl implementations, can you enlighten me by what you mean by "integrate the memory-context notion"? Does that mean they are not using palloc/pfree stuff? > > It's fairly difficult to get anywhere with standard leak-tracking tools, > since they don't know anything about palloc. What's worse, it is *not* > a bug for a routine to palloc space it never pfrees, if it knows that > it's palloc'ing in a short-lived memory context. The fact that a > context may be released with much still-allocated memory in it is not a > bug but a feature; but try teaching that to any standard leak checker... > > regards, tom lane Well, the thing that really got my attention is that dmalloc is reporting frees on null pointers. While that may be safe on specific platforms, IIRC, it's actually undefined. Again, this is as reported by dmalloc so I've yet to actually track it down to determine if it's possibly something else going on (magic voodoo of some heap manager, etc). Greg ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Memory leaks
Petru Paler <[EMAIL PROTECTED]> writes: > valgrind is a great tool I used -- didn't get the time to try it out on > Postgres yet, though. Besides leaks, it also catches uninitialized > variable access and stuff like that. I've used Valgrind with PostgreSQL a little bit, and it's been fairly useful (I used it to fix some memory leaks in psql and pg_dump and a couple of uninitialized memory accesses in the backend). If you want to use it on the backend, you'll need to stop postgres from clobbering ARGV (as this causes valgrind problems, for some reason) -- add '-DPS_USE_NONE -UPS_USE_CLOBBER_ARGV' to CFLAGS. I mentioned it to the author of valgrind, but IIRC he didn't mention any plans to change this behavior. Cheers, Neil -- Neil Conway <[EMAIL PROTECTED]> || PGP Key ID: DB3C29FC ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On Wed, 2002-10-23 at 03:09, Tom Lane wrote: > It's fairly difficult to get anywhere with standard leak-tracking tools, > since they don't know anything about palloc. What's worse, it is *not* > a bug for a routine to palloc space it never pfrees, if it knows that > it's palloc'ing in a short-lived memory context. The fact that a > context may be released with much still-allocated memory in it is not a > bug but a feature; but try teaching that to any standard leak checker... Seems that Valgrind should have no problems with it, as it tracks actual usage of _memory_ (down to single bits :)) , not malloc/free. See: http://developer.kde.org/~sewardj/ --- Hannu ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Memory leaks
Greg Copeland <[EMAIL PROTECTED]> writes: > I've started playing a little with Postgres to determine if there were > memory leaks running around. After some very brief checking, I'm > starting[1] to think that the answer is yes. Has anyone already gone > through a significant effort to locate and eradicate memory leaks? Yes, this has been dealt with before. Have you read src/backend/utils/mmgr/README? AFAIK the major problems these days are not in the backend as a whole, but in the lesser-used PL language modules (cf recent discussions). plpgsql has some issues too, I suspect, but not as bad as pltcl etc. Possibly the best answer is to integrate the memory-context notion into those modules; if they did most of their work in a temp context that could be freed once per PL statement or so, the problems would pretty much go away. It's fairly difficult to get anywhere with standard leak-tracking tools, since they don't know anything about palloc. What's worse, it is *not* a bug for a routine to palloc space it never pfrees, if it knows that it's palloc'ing in a short-lived memory context. The fact that a context may be released with much still-allocated memory in it is not a bug but a feature; but try teaching that to any standard leak checker... regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Memory leaks
On Tue, Oct 22, 2002 at 04:16:16PM -0500, Greg Copeland wrote: > I've started playing a little with Postgres to determine if there were > memory leaks running around. After some very brief checking, I'm > starting[1] to think that the answer is yes. Has anyone already gone > through a significant effort to locate and eradicate memory leaks? Is > this done periodically? If so, what tools are others using? I'm > currently using dmalloc for my curiosity. valgrind is a great tool I used -- didn't get the time to try it out on Postgres yet, though. Besides leaks, it also catches uninitialized variable access and stuff like that. Petru ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
