Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-27 Thread Gaddam Sai Ram
Thanks for your responses Michael and Amit Kapila,



Yes, we have included your suggestions in our code and started 
testing to reproduce the same issue. In case we encounter this issue again we 
will get back here.



Regards

G. Sai Ram










Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-26 Thread Amit Kapila
On Wed, Oct 25, 2017 at 8:04 PM, Gaddam Sai Ram
 wrote:
> Thanks for the response,
>
> Can you check if CurTransactionContext is valid at that point?
>
>
> Any Postgres function to check if CurTransactionContext is valid or not?
>
> To see, if this problem is related to CurTransactionContext, can you try to
> populate the list in TopMemoryContext and see if that works.
>
>
> Did you mean TopTransactionContext?
>

No, I mean what I have written.  I suspect in your case
TopTransactionContext will be same as CurTransactionContext because
you don't have any subtransaction.

> As of now, we don't free our dlist. We solely depend on Postgres to free our
> dlist while it frees the TopTransactionContext. But if we do allocate in
> TopMemoryContext, we need to take care of freeing our allocations.
>

Can't we do it temporarily to test?  I am not suggesting to make this
a permanent change rather a way to see the reason of the problem.

> And one more issue is, we found this bug once in all the testing we did. So
> trying to replay this bug seems very difficult.
>

Oh, then it is tricky.  I think there is a good chance that this is
some of your application issues where you probably haven't used memory
context as required, so probably you need to figure out a way to
reproduce this issue, otherwise, it might be difficult to track down
the actual cause.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-25 Thread Michael Paquier
On Wed, Oct 25, 2017 at 7:34 AM, Gaddam Sai Ram
 wrote:
> Thanks for the response,
>
> Can you check if CurTransactionContext is valid at that point?
>
>
> Any Postgres function to check if CurTransactionContext is valid or not?

Assert(MemoryContextIsValid(CurTransactionContext));
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-25 Thread Gaddam Sai Ram
Thanks for the response,


Can you check if CurTransactionContext is valid at that point? 








Any Postgres function to check if CurTransactionContext is valid or not?


To see, if this problem is related to CurTransactionContext, can you try to 
populate the list in TopMemoryContext and see if that works.








Did you mean TopTransactionContext? 
As of now, we don't free our dlist. We solely depend on Postgres to free our 
dlist while it frees the TopTransactionContext. But if we do allocate in 
TopMemoryContext, we need to take care of freeing our allocations.


And one more issue is, we found this bug once in all the testing we did. So 
trying to replay this bug seems very difficult.



Regards,

G. Sai Ram






Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-25 Thread Amit Kapila
On Tue, Oct 24, 2017 at 7:42 PM, Gaddam Sai Ram  wrote:

>
> Hello people,
>
> We are implementing in-memory index. As a part of that, during
> index callbacks, under CurTransactionContext, we cache all the DMLs of a
> transaction in dlist(postgres's doubly linked list).
> We registered transaction callback, and in transaction
> pre-commit event(XACT_EVENT_PRE_COMMIT), we iterate through all cached
> DMLs(dlist) and populate in my in-memory data structure.
>
>--> For detailed explanation of our implementation, please look
> into below thread.
>https://www.postgresql.org/message-id/15f4ea99b34.
> e69a4e1b633.8937474039794097334%40zohocorp.com
>
>--> It was working fine until I was greeted with a segmentation
> fault while accessing dlist!
>
>--> On further examining I found that dlist head is not NULL
> and it is pointing to some address, but the container I extracted is
> pointing to 0x7f7f7f7f7f7f7f5f, and I was not able to access any members in
> my container.
>
>--> https://wiki.postgresql.org/wiki/Developer_FAQ#Why_
> are_my_variables_full_of_0x7f_bytes.3F
> From the above wiki, we came to a conclusion that the memory
> context in which that dlist was present was freed.
> And yes CLOBBER_FREED_MEMORY is enabled by passing
> --enable-cassert to enable asserts in my code.
>
>--> Normally the memory context inside XACT_EVENT_PRE_COMMIT is
> TopTransactionContext but in this particular case, the memory context was
> 'MessageContext'. Little unusual! Not sure why!
>
>--> So basically CurTransactionContext shouldn't get freed
> before transaction COMMIT.
>--> So what has actually happened here??? Kindly help us with
> your insights!
>
>
Can you check if CurTransactionContext is valid at that point?  To see, if
this problem is related to CurTransactionContext, can you try to populate
the list in TopMemoryContext and see if that works.


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-24 Thread Gaddam Sai Ram




This sounds broken on its face --- if you want stuff to survive to 

top-level commit, you need to keep it in TopTransactionContext. 

CurTransactionContext might be a subtransaction's context that will 

go away at subtransaction commit/abort. 







https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README



>From the above README:



CurTransactionContext --- this holds data that has to survive until the end

of the current transaction, and in particular will be needed at top-level

transaction commit.  When we are in a top-level transaction this is the same

as TopTransactionContext, but in subtransactions it points to a child context.

It is important to understand that if a subtransaction aborts, its

CurTransactionContext is thrown away after finishing the abort processing;

but a committed subtransaction's CurTransactionContext is kept until top-level

commit (unless of course one of the intermediate levels of subtransaction

aborts).  This ensures that we do not keep data from a failed subtransaction

longer than necessary.  Because of this behavior, you must be careful to clean

up properly during subtransaction abort --- the subtransaction's state must be

delinked from any pointers or lists kept in upper transactions, or you will

have dangling pointers leading to a crash at top-level commit.  An example of

data kept here is pending NOTIFY messages, which are sent at top-level commit,

but only if the generating subtransaction did not abort.



-- So even if sub-transaction is committed, subtransaction's 
CurTransactionContext is kept until top-level commit.

-- If sub-transaction is aborted, we handled(clearing the data) it via 
RegisterSubXactCallback().



And in our particular use case(which gave segmentation fault), we didn't issue 
any sub-transaction. It was a single insert transaction.

Even then it resulted in some kind of context free.




Regards

G. Sai Ram








Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-24 Thread Tom Lane
Gaddam Sai Ram  writes:
> We are implementing in-memory index. As a part of that, during 
> index callbacks, under CurTransactionContext, we cache all the DMLs of a 
> transaction in dlist(postgres's doubly linked list).
> We registered transaction callback, and in transaction pre-commit 
> event(XACT_EVENT_PRE_COMMIT), we iterate through all cached DMLs(dlist) and 
> populate in my in-memory data structure.

This sounds broken on its face --- if you want stuff to survive to
top-level commit, you need to keep it in TopTransactionContext.
CurTransactionContext might be a subtransaction's context that will
go away at subtransaction commit/abort.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers