Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???
Thanks for your responses Michael and Amit Kapila, Yes, we have included your suggestions in our code and started testing to reproduce the same issue. In case we encounter this issue again we will get back here. Regards G. Sai Ram
Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???
Thanks for the response, Can you check if CurTransactionContext is valid at that point? Any Postgres function to check if CurTransactionContext is valid or not? To see, if this problem is related to CurTransactionContext, can you try to populate the list in TopMemoryContext and see if that works. Did you mean TopTransactionContext? As of now, we don't free our dlist. We solely depend on Postgres to free our dlist while it frees the TopTransactionContext. But if we do allocate in TopMemoryContext, we need to take care of freeing our allocations. And one more issue is, we found this bug once in all the testing we did. So trying to replay this bug seems very difficult. Regards, G. Sai Ram
Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???
This sounds broken on its face --- if you want stuff to survive to top-level commit, you need to keep it in TopTransactionContext. CurTransactionContext might be a subtransaction's context that will go away at subtransaction commit/abort. https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README >From the above README: CurTransactionContext --- this holds data that has to survive until the end of the current transaction, and in particular will be needed at top-level transaction commit. When we are in a top-level transaction this is the same as TopTransactionContext, but in subtransactions it points to a child context. It is important to understand that if a subtransaction aborts, its CurTransactionContext is thrown away after finishing the abort processing; but a committed subtransaction's CurTransactionContext is kept until top-level commit (unless of course one of the intermediate levels of subtransaction aborts). This ensures that we do not keep data from a failed subtransaction longer than necessary. Because of this behavior, you must be careful to clean up properly during subtransaction abort --- the subtransaction's state must be delinked from any pointers or lists kept in upper transactions, or you will have dangling pointers leading to a crash at top-level commit. An example of data kept here is pending NOTIFY messages, which are sent at top-level commit, but only if the generating subtransaction did not abort. -- So even if sub-transaction is committed, subtransaction's CurTransactionContext is kept until top-level commit. -- If sub-transaction is aborted, we handled(clearing the data) it via RegisterSubXactCallback(). And in our particular use case(which gave segmentation fault), we didn't issue any sub-transaction. It was a single insert transaction. Even then it resulted in some kind of context free. Regards G. Sai Ram
[HACKERS] CurTransactionContext freed before transaction COMMIT ???
Hello people,
We are implementing in-memory index. As a part of that, during
index callbacks, under CurTransactionContext, we cache all the DMLs of a
transaction in dlist(postgres's doubly linked list).
We registered transaction callback, and in transaction pre-commit
event(XACT_EVENT_PRE_COMMIT), we iterate through all cached DMLs(dlist) and
populate in my in-memory data structure.
-- For detailed explanation of our implementation, please look
into below thread.
https://www.postgresql.org/message-id/15f4ea99b34.e69a4e1b633.8937474039794097334%40zohocorp.com
-- It was working fine until I was greeted with a segmentation
fault while accessing dlist!
-- On further examining I found that dlist head is not NULL and
it is pointing to some address, but the container I extracted is pointing to
0x7f7f7f7f7f7f7f5f, and I was not able to access any members in my container.
--
https://wiki.postgresql.org/wiki/Developer_FAQ#Why_are_my_variables_full_of_0x7f_bytes.3F
From the above wiki, we came to a conclusion that the memory
context in which that dlist was present was freed.
And yes CLOBBER_FREED_MEMORY is enabled by passing --enable-cassert
to enable asserts in my code.
-- Normally the memory context inside XACT_EVENT_PRE_COMMIT is
TopTransactionContext but in this particular case, the memory context was
'MessageContext'. Little unusual! Not sure why!
-- So basically CurTransactionContext shouldn't get freed before
transaction COMMIT.
-- So what has actually happened here??? Kindly help us with
your insights!
For your reference, a code snippet of our implementation is pasted below:
Sample code snippet:
/*** Code snippet starts ***/
dlist_head DMLInfo = {{NULL, NULL}};
Insert_callback()
{
old_context = MemoryContextSwitchTo(CurTransactionContext);
GraphIndex_InsertInfo *current;
current = (GraphIndex_InsertInfo *)palloc(sizeof(GraphIndex_InsertInfo));
current-tableOid = tableOid;
current-subTransactionID = GetCurrentSubTransactionId();
current-colValue = (long)values[0]; // Varies with column type
current-ctid = ctid;
dlist_push_head(DMLInfo, current-list_node);
MemoryContextSwitchTo(old_context);
return TRUE;
}
static void GraphIndex_xactCallback(XactEvent event, void *arg)
{
GraphIndex_Table *tableInfo;
GraphIndex_InsertInfo *current;
dlist_iter iter;
switch (event)
{
case XACT_EVENT_PRE_PREPARE:
case XACT_EVENT_PRE_COMMIT:
PG_TRY();
{
if(DMLInfo.head.next)
{
dlist_reverse_foreach(iter, DMLInfo)
{
current =
dlist_container(GraphIndex_InsertInfo, list_node, iter.cur);
DMLProcessingFunction(current); // This is
giving me the ERROR (while accessing members of current)
}
DMLInfo.head.next = NULL;
}
}
PG_CATCH();
{
/* Do cleanup */
}
PG_END_TRY();
break;
case XACT_EVENT_ABORT:
case XACT_EVENT_PARALLEL_ABORT:
/* No cleanup Necessary(No structure created) */
break;
default:
return;
}
}
/*** Code snippet ends ***/
Regards
G. Sai Ram
[HACKERS] Help needed regarding DSA based in-memory index!
Hi Munro, Thanks for cautioning us about possible memory leaks(during error cases) incase of long-lived DSA segements(have a look in below thread for more details). https://www.postgresql.org/message-id/CAEepm%3D3c4WAtSQG4tAF7Y_VCnO5cKh7KuFYZhpKbwGQOF%3DdZ4A%40mail.gmail.com Actually we are following an approach to avoid this DSA memory leaks. Let me explain our implementation and please validate and correct us in-case we miss anything. Implementation: Basically we have to put our index data into memory (Index Column Value Vs Ctid) which we get in aminsert callback function. Coming to the implementation, in aminsert Callback function, We Switch to CurTransactionContext Cache the DMLs of a transaction into dlist(global per process) Even if different clients work parallel, it won't be a problem because every client gets one dlist in separate process and it'll have it's own CurTransactionContext We have registered transaction callback (using RegisterXactCallback() function). And during event pre-commit(XACT_EVENT_PRE_COMMIT), we populate all the transaction specific DMLs (from dlist) into our in-memory index(DSA) obviously inside PG_TRY/PG_CATCH block. In case we got some errors(because of dsa_allocate() or something else) while processing dlist(while populating in-memory index), we cleanup the DSA memory in PG_CATCH block that is allocated/used till that point. During other error cases, typically transactions gets aborted and PRE_COMMIT event is not called and hence we don't touch DSA at that time. Hence no need to bother about leaks. Even sub transaction case is handled with sub transaction callbacks. CurTransactionContext(dlist basically) is automatically cleared after that particular transaction. I want to know if this approach is good and works well in all cases. Kindly provide your feedback on this. Regards G. Sai Ram
Re: [HACKERS] Error: dsa_area could not attach to a segment that has been freed
Hi Thomas, Thanks for cautioning us about possible memory leaks(during error cases) incase of long-lived DSA segements. Actually we are following an approach to avoid this DSA memory leaks. Let me explain our implementation and please validate and correct us in-case we miss anything. Implementation: Basically we have to put our index data into memory (Index Column Value Vs Ctid) which we get in aminsert callback function. Coming to the implementation, in aminsert Callback function, We Switch to CurTransactionContext Cache the DMLs of a transaction into dlist(global per process) Even if different clients work parallel, it won't be a problem because every client gets one dlist in separate process and it'll have it's own CurTransactionContext We have registered transaction callback (using RegisterXactCallback() function). And during event pre-commit(XACT_EVENT_PRE_COMMIT), we populate all the transaction specific DMLs (from dlist) into our in-memory index(DSA) obviously inside PG_TRY/PG_CATCH block. In case we got some errors(because of dsa_allocate() or something else) while processing dlist(while populating in-memory index), we cleanup the DSA memory in PG_CATCH block that is allocated/used till that point. During other error cases, typically transactions gets aborted and PRE_COMMIT event is not called and hence we don't touch DSA at that time. Hence no need to bother about leaks. Even sub transaction case is handled with sub transaction callbacks. CurTransactionContext(dlist basically) is automatically cleared after that particular transaction. I want to know if this approach is good and works well in all cases. Kindly provide your feedback on this. Regards G. Sai Ram On Wed, 20 Sep 2017 14:25:43 +0530 Thomas Munro thomas.mu...@enterprisedb.com wrote On Wed, Sep 20, 2017 at 6:14 PM, Gaddam Sai Ram gaddamsaira...@zohocorp.com wrote: Thank you very much! That fixed my issue! :) I was in an assumption that pinning the area will increase its lifetime but yeah after taking memory context into consideration its working fine! So far the success rate in confusing people who first try to make long-lived DSA areas and DSM segments is 100%. Basically, this is all designed to ensure automatic cleanup of resources in short-lived scopes. Good luck for your graph project. I think you're going to have to expend a lot of energy trying to avoid memory leaks if your DSA lives as long as the database cluster, since error paths won't automatically free any memory you allocated in it. Right now I don't have any particularly good ideas for mechanisms to deal with that. PostgreSQL C has exception-like error handling, but doesn't (and probably can't) have a language feature like scoped destructors from C++. IMHO exceptions need either destructors or garbage collection to keep you sane. There is a kind of garbage collection for palloc'd memory and also for other resources like file handles, but if you're using a big long lived DSA area you have nothing like that. You can use PG_TRY/PG_CATCH very carefully to clean up, or (probably better) you can try to make sure that all your interaction with shared memory is no-throw (note that that means using dsa_allocate_extended(x, DSA_ALLOC_NO_OOM), because dsa_allocate itself can raise errors). The first thing I'd try would probably be to keep all shmem-allocating code in as few routines as possible, and use only no-throw operations in the 'critical' regions of them, and maybe look into some kind of undo log of things to free or undo in case of error to manage multi-allocation operations if that turned out to be necessary. -- Thomas Munro http://www.enterprisedb.com
Re: [HACKERS] Error: dsa_area could not attach to a segment that has been freed
Thank you very much! That fixed my issue! :) I was in an assumption that pinning the area will increase its lifetime but yeah after taking memory context into consideration its working fine! Regards G. Sai Ram On Wed, 20 Sep 2017 11:16:19 +0530 Thomas Munro thomas.mu...@enterprisedb.com wrote On Fri, Sep 15, 2017 at 7:51 PM, Gaddam Sai Ram gaddamsaira...@zohocorp.com wrote: As i'm pinning the dsa mapping after attach, it has to stay through out the backend session. But not sure why its freed/corrupted. Kindly help me in fixing this issue. Attached the copy of my extension, which will reproduce the same issue. Your DSA area is pinned and the mapping is pinned, but there is one more thing that goes away automatically unless you nail it to the table: the backend-local dsa_area object which dsa_create() and dsa_attach() return. That's allocated in the "current memory context", so if you do it from your procedure simple_udf_func without making special arrangements it gets automatically freed at end of transaction. If you're going to cache it for the whole life of the backend, you'd better make sure it's allocated in memory context that lives long enough. Where you have dsa_create() and dsa_attach() calls, try coding like this: MemoryContext old_context; old_context = MemoryContextSwitchTo(TopMemoryContext); area = dsa_create(...); MemoryContextSwitchTo(old_context); old_context = MemoryContextSwitchTo(TopMemoryContext); area = dsa_attach(...); MemoryContextSwitchTo(old_context); You'll need to #include "utils/memutils.h". -- Thomas Munro http://www.enterprisedb.com
[HACKERS] Re: Error: dsa_area could not attach to a segment that has been freed
Kindly help me with the above thread.. Thanks G. Sai Ram On Fri, 15 Sep 2017 13:21:33 +0530 Gaddam Sai Ram gaddamsaira...@zohocorp.com wrote Hello everyone, Based on the discussion in the below thread, I built a an extension using DSA(postgres-10 beta-3, linux machine). https://www.postgresql.org/message-id/flat/15cc91e836b.12362f6336537.3402818166899386386%40zohocorp.com#15cc91e836b.12362f6336537.3402818166899386...@zohocorp.com Use _PG_init and the shmem hook to reserve a little bit of traditional shared memory and initialise it to zero. This will be used just to share the DSA handle, but you can't actually create the DSA area in postmaster. In other words, this little bit of shared memory is for "discovery", since it can be looked up by name from any backend. Yes, I have created memory for DSA handle in shared memory, but not the actual DSA area. In each backend that wants to use your new in-memory index system, you need to be able to attach or create the DSA area on-demand. Perhaps you could have a get_my_shared_state() function (insert better name) that uses a static local variable to hold a pointer to some state. If it's NULL, you know you need to create the state. That should happen only once in each backend, the first time through the function. In that case you need to create or attach to the DSA area as appropriate, which you should wrap in LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE)/LWLockRelease(AddinShmemInitLock) to serialise the code block. First, look up the bit of traditional shared memory to see if there is a DSA handle published in it already. If there is you can attach. If there isn't, you are the first so you need to create, and publish the handle for others to attach to. Remember whatever state you need to remember, such as the dsa_area, in static local variables so that all future calls to get_my_shared_state() in that backend will be fast. Yes, the code is present in gstore_shmem.c(pfa) and the first process to use DSA will create the area, and rest all new processes will either attach it or if it is already attached, it will use the DSA area which is already pinned. == I have created a bgworker in pg_init and when it starts it will be the first process to access DSA, so it will create DSA area. == I have a small UDF function(simple_udf_func) which I call in a new backend(process). So it will attach the DSA area already created. == When I make a call to same UDF function again in the same process, since the area is already attached and pinned, I use the same area which I store in a global variable while attaching/creating. Here I get the problem... Error details: dsa_area could not attach to a segment that has been freed While examining in detail, I found this data. I used dsa_dump() for debugging and I found that during my error case, i get this log: dsa_area handle 1: max_total_segment_size: 0 total_segment_size: 0 refcnt: 0 pinned: f segment bins: segment bin 0 (at least -2147483648 contiguous pages free): Clearly, the data in my DSA area has been corrupted in latter case, but my bgworker continues to work proper with same dsa_area handle. At this stage, the dsa_dump() in my bgworker is as below: dsa_area handle 1814e630: max_total_segment_size: 18446744073709551615 total_segment_size: 1048576 refcnt: 3 pinned: t segment bins: segment bin 8 (at least 128 contiguous pages free): segment index 0, usable_pages = 253, contiguous_pages = 220, mapped at 0x7f0abbd58000 As i'm pinning the dsa mapping after attach, it has to stay through out the backend session. But not sure why its freed/corrupted. Kindly help me in fixing this issue. Attached the copy of my extension, which will reproduce the same issue. Regards G. Sai Ram
