Re: [HACKERS] Speedup twophase transactions

On Tue, Mar 28, 2017 at 3:10 PM, Michael Paquier wrote: > OK, done. I have just noticed that Simon has marked himself as a > committer of this patch 24 hours ago. For the archive's sake, this has been committed as 728bd991. Thanks Simon! -- Michael -- Sent via

Re: [HACKERS] Speedup twophase transactions

On Tue, Mar 28, 2017 at 3:08 PM, Nikhil Sontakke wrote: >> >> I don't have anything else to say about this patch, so should we mark >> that as ready for committer? There are still a couple of days left >> until the end of the CF, and quite a lot has happened, so this

Re: [HACKERS] Speedup twophase transactions

> > > I don't have anything else to say about this patch, so should we mark > that as ready for committer? There are still a couple of days left > until the end of the CF, and quite a lot has happened, so this could > get on time into PG10. > Yes, let's mark it "ready for committer". This

Re: [HACKERS] Speedup twophase transactions

On Tue, Mar 28, 2017 at 2:10 PM, Nikhil Sontakke wrote: > The latest patch looks good. By now doing a single scan of shmem two phase > data, we have removed the double loops in all the affected functions which > is good. Yup. > My only question is if the added call to

Re: [HACKERS] Speedup twophase transactions

Please ignore reports about errors in other tests. Seem spurious.. Regards, Nikhils On 28 March 2017 at 10:40, Nikhil Sontakke wrote: > Hi Micheal, > > The latest patch looks good. By now doing a single scan of shmem two phase > data, we have removed the double loops

Re: [HACKERS] Speedup twophase transactions

Hi Micheal, The latest patch looks good. By now doing a single scan of shmem two phase data, we have removed the double loops in all the affected functions which is good. My only question is if the added call to restoreTwoPhaseData() is good enough to handle all the 3 functions

Re: [HACKERS] Speedup twophase transactions

On Sun, Mar 26, 2017 at 4:50 PM, Nikhil Sontakke wrote: > I was away for a bit. I will take a look at this patch and get back to you > soon. No problem. Thanks for your time! -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Speedup twophase transactions

Thanks Michael, I was away for a bit. I will take a look at this patch and get back to you soon. Regards, Nikhils On 22 March 2017 at 07:40, Michael Paquier wrote: > On Fri, Mar 17, 2017 at 5:15 PM, Michael Paquier > wrote: > > On Fri,

Re: [HACKERS] Speedup twophase transactions

On Fri, Mar 17, 2017 at 5:15 PM, Michael Paquier wrote: > On Fri, Mar 17, 2017 at 5:00 PM, Nikhil Sontakke > wrote: >> Micheal, it looks like you are working on a final version of this patch? I >> will wait to review it from my end, then. > > I

Re: [HACKERS] Speedup twophase transactions

On Fri, Mar 17, 2017 at 5:00 PM, Nikhil Sontakke wrote: >> > >> > Ok, we can do that and then yes, RecoverPreparedTransaction() can just >> > have >> > one loop going through the shmem entries. BUT, we cannot ignore >> > "inredo"+"ondisk" entries. For such entries, we

Re: [HACKERS] Speedup twophase transactions

> > > > > Ok, we can do that and then yes, RecoverPreparedTransaction() can just > have > > one loop going through the shmem entries. BUT, we cannot ignore > > "inredo"+"ondisk" entries. For such entries, we will have to read and > > recover from the corresponding 2PC files. > > Yes. About other

Re: [HACKERS] Speedup twophase transactions

On Fri, Mar 17, 2017 at 4:42 PM, Nikhil Sontakke wrote: >> > >> > I don't think this will work. We cannot replace pg_twophase with shmem >> > entries + WAL pointers. This is because we cannot expect to have WAL >> > entries >> > around for long running prepared queries

Re: [HACKERS] Speedup twophase transactions

> > > > > I don't think this will work. We cannot replace pg_twophase with shmem > > entries + WAL pointers. This is because we cannot expect to have WAL > entries > > around for long running prepared queries which survive across > checkpoints. > > But at the beginning of recovery, we can mark

Re: [HACKERS] Speedup twophase transactions

> > Nikhil, do you mind if I try something like that? As we already know > what is the first XID when beginning redo via > ShmemVariableCache->nextXid it is possible to discard 2PC files that > should not be here. Yeah, that is ok. > What makes me worry is the control of the maximum > number

Re: [HACKERS] Speedup twophase transactions

On Thu, Mar 16, 2017 at 9:25 PM, Michael Paquier wrote: > On Thu, Mar 16, 2017 at 7:18 PM, Nikhil Sontakke > wrote: >>> + * * RecoverPreparedTransactions(), >>> StandbyRecoverPreparedTransactions() >>> + *and

Re: [HACKERS] Speedup twophase transactions

On Thu, Mar 16, 2017 at 7:18 PM, Nikhil Sontakke wrote: >> + * * RecoverPreparedTransactions(), >> StandbyRecoverPreparedTransactions() >> + *and PrescanPreparedTransactions() have been modified to go >> throug >> + *gxact->inredo entries that have

Re: [HACKERS] Speedup twophase transactions

> > + * * RecoverPreparedTransactions(), StandbyRecoverPreparedTransact > ions() > + *and PrescanPreparedTransactions() have been modified to go > throug > + *gxact->inredo entries that have not made to disk yet. > > It seems to me that there should be an initial scan of

Re: [HACKERS] Speedup twophase transactions

On Wed, Mar 15, 2017 at 4:48 PM, Nikhil Sontakke wrote: >> boolvalid; /* TRUE if PGPROC entry is in proc array >> */ >> boolondisk; /* TRUE if prepare state file is on disk >> */ >> + boolinredo; /* TRUE if entry

Re: [HACKERS] Speedup twophase transactions

> Thanks for the new version. Let's head toward a final patch. > > :-) > > > > Have added a new function to do this now. It reads either from disk or > > shared memory and produces error/log messages accordingly as well now. > > And that's ProcessTwoPhaseBufferAndReturn in the patch. >

Re: [HACKERS] Speedup twophase transactions

On Sat, Mar 11, 2017 at 7:26 PM, Nikhil Sontakke wrote: > Hi David and Michael, >> It would be great to get this thread closed out after 14 months and many >> commits. > > PFA, latest patch which addresses Michael's comments. Thanks for the new version. Let's head toward

Re: [HACKERS] Speedup twophase transactions

Hi David and Michael, > It would be great to get this thread closed out after 14 months and many > commits. > > PFA, latest patch which addresses Michael's comments. twophase.c: In function ‘PrepareRedoAdd’: > twophase.c:2539:20: warning: variable ‘gxact’ set but not used >

Re: [HACKERS] Speedup twophase transactions

Nikhil, On 2/27/17 12:19 AM, Nikhil Sontakke wrote: > Hi Michael, > > Thanks for taking a look at the patch. > > twophase.c: In function ‘PrepareRedoAdd’: > twophase.c:2539:20: warning: variable ‘gxact’ set but not used > [-Wunused-but-set-variable] > GlobalTransaction gxact;

Re: [HACKERS] Speedup twophase transactions

Hi Michael, Thanks for taking a look at the patch. > twophase.c: In function ‘PrepareRedoAdd’: > twophase.c:2539:20: warning: variable ‘gxact’ set but not used > [-Wunused-but-set-variable] > GlobalTransaction gxact; > There is a warning at compilation. > > Will fix. > The comment at the

Re: [HACKERS] Speedup twophase transactions

On Thu, Feb 2, 2017 at 3:07 PM, Nikhil Sontakke wrote: >>> Yeah. Was thinking about this yesterday. How about adding entries in >>> TwoPhaseState itself (which become valid later)? Only if it does not >>> cause a lot of code churn. >> >> That's possible as well, yes. > >

Re: [HACKERS] Speedup twophase transactions

On Wed, Feb 22, 2017 at 7:03 AM, Alvaro Herrera wrote: > I have pushed the TAP test file, which is already passing, at least for > me. Let's see what the buildfarm says. Thanks. I still have on my sheet to look at the latest 2PC patch. -- Michael -- Sent via

Re: [HACKERS] Speedup twophase transactions

I have pushed the TAP test file, which is already passing, at least for me. Let's see what the buildfarm says. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Speedup twophase transactions

On Wed, Feb 1, 2017 at 3:29 AM, Nikhil Sontakke wrote: >>> Shouldn’t the singular part of the message above be: >>> "%u two-phase state file was written for a long-running prepared >>> transaction" >>> >>> But, then, English is not my native language, so I might be off

Re: [HACKERS] Speedup twophase transactions

>> Yeah. Was thinking about this yesterday. How about adding entries in >> TwoPhaseState itself (which become valid later)? Only if it does not >> cause a lot of code churn. > > That's possible as well, yes. PFA a patch which does the above. It re-uses the TwoPhaseState gxact entries to track 2PC

Re: [HACKERS] Speedup twophase transactions

>> Shouldn’t the singular part of the message above be: >> "%u two-phase state file was written for a long-running prepared transaction" >> >> But, then, English is not my native language, so I might be off here :-) > > If there's one file per long-running prepared transaction, which I > think is

Re: [HACKERS] Speedup twophase transactions

On Mon, Jan 23, 2017 at 7:00 AM, Nikhil Sontakke wrote: > 4) Minor nit-pick on existing code. > > (errmsg_plural("%u two-phase state file was written " > "for > long-running prepared transactions", >

Re: [HACKERS] Speedup twophase transactions

On Tue, Jan 31, 2017 at 2:58 PM, Nikhil Sontakke wrote: >>> CheckPointTwoPhase() in (5) does not sync this prepared transaction >>> because the checkpointer's KnownPreparedList is empty. >> >> And that's why this needs to be stored in shared memory with a number >> of

Re: [HACKERS] Speedup twophase transactions

>> CheckPointTwoPhase() in (5) does not sync this prepared transaction >> because the checkpointer's KnownPreparedList is empty. > > And that's why this needs to be stored in shared memory with a number > of elements made of max_prepared_xacts... Yeah. Was thinking about this yesterday. How about

Re: [HACKERS] Speedup twophase transactions

On Tue, Jan 31, 2017 at 2:34 PM, Nikhil Sontakke wrote: >>> I wonder what's the best location for this in the common case when we >>> do shutdown of standby. We could add code in XLOG_CHECKPOINT_SHUTDOWN >>> and XLOG_CHECKPOINT_ONLINE xlog_redo code path. >> >>

Re: [HACKERS] Speedup twophase transactions

>> Having CheckPointTwoPhase() do the flush would mean shifting the data >> from KnownPreparedList into TwoPhaseState shmem. > > Er, no. For CheckPointTwoPhase(), at recovery what needs to be done is > to have all the entries in KnownPreparedList() flushed to disk and > have those entries removed

Re: [HACKERS] Speedup twophase transactions

On Tue, Jan 31, 2017 at 3:45 AM, Nikhil Sontakke wrote: >> --- a/src/backend/access/transam/xlog.c >> +++ b/src/backend/access/transam/xlog.c >> @@ -9573,6 +9573,7 @@ xlog_redo(XLogReaderState *record) >> (errmsg("unexpected timeline ID %u (should be

Re: [HACKERS] Speedup twophase transactions

> --- a/src/backend/access/transam/xlog.c > +++ b/src/backend/access/transam/xlog.c > @@ -9573,6 +9573,7 @@ xlog_redo(XLogReaderState *record) > (errmsg("unexpected timeline ID %u (should be %u) > in checkpoint record", > checkPoint.ThisTimeLineID,

Re: [HACKERS] Speedup twophase transactions

>>> The xact_redo code will add prepared transactions to the >>> KnownPreparedList in memory. Earlier it used to create the on-disk 2PC >>> file. >>> >>> At standby promote, the surviving (yet uncommitted) prepared >>> transactions from KnownPreparedList need to be persisted, right? >> >> I don't

Re: [HACKERS] Speedup twophase transactions

On Fri, Jan 27, 2017 at 8:23 PM, Simon Riggs wrote: > On 27 January 2017 at 11:01, Nikhil Sontakke wrote: >> The xact_redo code will add prepared transactions to the >> KnownPreparedList in memory. Earlier it used to create the on-disk 2PC >> file.

Re: [HACKERS] Speedup twophase transactions

On 27 January 2017 at 11:01, Nikhil Sontakke wrote: > On 27 January 2017 at 15:37, Simon Riggs wrote: >> On 27 January 2017 at 09:59, Nikhil Sontakke wrote: > But, I put the recovery process and the checkpointer

Re: [HACKERS] Speedup twophase transactions

On 27 January 2017 at 15:37, Simon Riggs wrote: > On 27 January 2017 at 09:59, Nikhil Sontakke wrote: But, I put the recovery process and the checkpointer process of the standby under gdb with breakpoints on these functions, but both did

Re: [HACKERS] Speedup twophase transactions

On 27 January 2017 at 09:59, Nikhil Sontakke wrote: >>> But, I put the recovery process and the checkpointer process of the >>> standby under gdb with breakpoints on these functions, but both did >>> not hit CreateRestartPoint() as well as CheckPointGuts() when I issued

Re: [HACKERS] Speedup twophase transactions

>> But, I put the recovery process and the checkpointer process of the >> standby under gdb with breakpoints on these functions, but both did >> not hit CreateRestartPoint() as well as CheckPointGuts() when I issued >> a promote :-| > > No end-of-recovery checkpoints happen at promotion since 9.3.

Re: [HACKERS] Speedup twophase transactions

On Thu, Jan 26, 2017 at 5:02 PM, Nikhil Sontakke wrote: >>> Umm, AFAICS, CheckPointTwoPhase() does not get called in the "standby >>> promote" code path. >> >> CreateRestartPoint() calls it via CheckPointGuts() while in recovery. > > May be that I am missing something. >

Re: [HACKERS] Speedup twophase transactions

>> Umm, AFAICS, CheckPointTwoPhase() does not get called in the "standby >> promote" code path. > > CreateRestartPoint() calls it via CheckPointGuts() while in recovery. May be that I am missing something. But, I put the recovery process and the checkpointer process of the standby under gdb with

Re: [HACKERS] Speedup twophase transactions

> On 26 Jan 2017, at 10:34, Michael Paquier wrote: > > On Thu, Jan 26, 2017 at 4:09 PM, Nikhil Sontakke > wrote: >>> I look at this patch from you and that's present for me: >>>

Re: [HACKERS] Speedup twophase transactions

On Thu, Jan 26, 2017 at 4:09 PM, Nikhil Sontakke wrote: >>I look at this patch from you and that's present for me: >>https://www.postgresql.org/message-id/CAMGcDxf8Bn9ZPBBJZba9wiyQq->Qk5uqq=vjomnrnw5s+fks...@mail.gmail.com > >> --- a/src/backend/access/transam/xlog.c >>

Re: [HACKERS] Speedup twophase transactions

>I look at this patch from you and that's present for me: >https://www.postgresql.org/message-id/CAMGcDxf8Bn9ZPBBJZba9wiyQq->Qk5uqq=vjomnrnw5s+fks...@mail.gmail.com > --- a/src/backend/access/transam/xlog.c > +++ b/src/backend/access/transam/xlog.c > @@ -9573,6 +9573,7 @@

Re: [HACKERS] Speedup twophase transactions

On Thu, Jan 26, 2017 at 1:38 PM, Nikhil Sontakke wrote: >> We should really try to do things right now, or we'll never come back >> to it. 9.3 (if my memory does not fail me?) has reduced the time to do >> promotion by removing the need of the end-of-recovery checkpoint,

Re: [HACKERS] Speedup twophase transactions

>> The question remains whether saving off a few fsyncs/reads for these >> long-lived prepared transactions is worth the additional code churn. >> Even if we add code to go through the KnownPreparedList, we still will >> have to go through the other on-disk 2PC transactions anyways. So, >> maybe

Re: [HACKERS] Speedup twophase transactions

On Wed, Jan 25, 2017 at 11:55 PM, Nikhil Sontakke wrote: >> We are talking about the recovery/promote code path. Specifically this >> call to KnownPreparedRecreateFiles() in PrescanPreparedTransactions(). >> >> We write the files to disk and they get immediately read up

Re: [HACKERS] Speedup twophase transactions

> We are talking about the recovery/promote code path. Specifically this > call to KnownPreparedRecreateFiles() in PrescanPreparedTransactions(). > > We write the files to disk and they get immediately read up in the > following code. We could not write the files to disk and read >

Re: [HACKERS] Speedup twophase transactions

> Thanks for review, Nikhil and Michael. > > I don’t follow here. We are moving data away from WAL to files on checkpoint > because after checkpoint > there is no guaranty that WAL segment with our prepared tx will be still > available. > We are talking about the recovery/promote code path.

Re: [HACKERS] Speedup twophase transactions

> On 24 Jan 2017, at 09:42, Michael Paquier wrote: > > On Mon, Jan 23, 2017 at 9:00 PM, Nikhil Sontakke > wrote: >> Speeding up recovery or failover activity via a faster promote is a >> desirable thing. So, maybe, we should look at teaching

Re: [HACKERS] Speedup twophase transactions

>> Speeding up recovery or failover activity via a faster promote is a >> desirable thing. So, maybe, we should look at teaching the relevant >> code about using "KnownPreparedList"? I know that would increase the >> size of this patch and would mean more testing, but this seems to be >> last

Re: [HACKERS] Speedup twophase transactions

On Mon, Jan 23, 2017 at 9:00 PM, Nikhil Sontakke wrote: > Hi Stas, > >> >> So, here is brand new implementation of the same thing. >> >> Now instead of creating pgproc entry for prepared transaction during >> recovery, >> I just store recptr/xid correspondence in

Re: [HACKERS] Speedup twophase transactions

Hi Stas, > > So, here is brand new implementation of the same thing. > > Now instead of creating pgproc entry for prepared transaction during recovery, > I just store recptr/xid correspondence in separate 2L-list and > deleting entries in that > list if redo process faced commit/abort. In case of

Re: [HACKERS] Speedup twophase transactions

On Tue, Dec 27, 2016 at 12:59 PM, Stas Kelvich wrote: > Standard config with increased shared_buffers. I think the most significant > impact on the recovery speed here is on the client side, namely time between > prepare and commit. Right now I’m using pgbench script

Re: [HACKERS] Speedup twophase transactions

> On 22 Dec 2016, at 05:35, Michael Paquier wrote: > > True. The more spread the checkpoints and 2PC files, the more risk to > require access to disk. Memory's cheap anyway. What was the system > memory? How many checkpoints did you trigger for how many 2PC files >

Re: [HACKERS] Speedup twophase transactions

On Thu, Dec 22, 2016 at 7:35 AM, Michael Paquier wrote: > On Wed, Dec 21, 2016 at 10:37 PM, Stas Kelvich > wrote: >> ISTM your reasoning about filesystem cache applies here as well, but just >> without spending time on file creation. > >

Re: [HACKERS] Speedup twophase transactions

On Wed, Dec 21, 2016 at 10:37 PM, Stas Kelvich wrote: > On 21 Dec 2016, at 19:56, Michael Paquier wrote: >> That's indeed way simpler than before. Have you as well looked at the >> most simple approach discussed? That would be just roughly

Re: [HACKERS] Speedup twophase transactions

> On 21 Dec 2016, at 19:56, Michael Paquier wrote: > > That's indeed way simpler than before. Have you as well looked at the > most simple approach discussed? That would be just roughly replacing > the pg_fsync() calls currently in RecreateTwoPhaseFile() by a save >

Re: [HACKERS] Speedup twophase transactions

On Fri, Dec 16, 2016 at 8:00 PM, Stas Kelvich wrote: > So, here is brand new implementation of the same thing. > > Now instead of creating pgproc entry for prepared transaction during > recovery, > I just store recptr/xid correspondence in separate 2L-list and deleting >

Re: [HACKERS] Speedup twophase transactions

On Sat, Dec 17, 2016 at 07:38:46PM -0500, Bruce Momjian wrote: > As Andres already stated, the problem is that there is a text/plain and > text/html of the same email, and the diff is _inside_ the multipa/mixed > HTML block. I think it needs to be outside on its own. Mutt shows text/plain by

Re: [HACKERS] Speedup twophase transactions

On Sat, Dec 17, 2016 at 03:47:34PM -0800, Andres Freund wrote: > On 2016-12-17 15:30:08 -0800, David Fetter wrote: > > On Sat, Dec 17, 2016 at 05:54:04PM -0500, Bruce Momjian wrote: > > > On Sun, Dec 18, 2016 at 07:41:50AM +0900, Michael Paquier wrote: > > > > On Sun, Dec 18, 2016 at 6:42 AM,

Re: [HACKERS] Speedup twophase transactions

On 2016-12-17 15:30:08 -0800, David Fetter wrote: > On Sat, Dec 17, 2016 at 05:54:04PM -0500, Bruce Momjian wrote: > > On Sun, Dec 18, 2016 at 07:41:50AM +0900, Michael Paquier wrote: > > > On Sun, Dec 18, 2016 at 6:42 AM, Bruce Momjian wrote: > > > > Uh, did you mean to

Re: [HACKERS] Speedup twophase transactions

On Sat, Dec 17, 2016 at 05:54:04PM -0500, Bruce Momjian wrote: > On Sun, Dec 18, 2016 at 07:41:50AM +0900, Michael Paquier wrote: > > On Sun, Dec 18, 2016 at 6:42 AM, Bruce Momjian wrote: > > > Uh, did you mean to attached patch here? > > > > Strange. I can confirm that I have

Re: [HACKERS] Speedup twophase transactions

> On 18 Dec 2016, at 01:54, Bruce Momjian wrote: > > On Sun, Dec 18, 2016 at 07:41:50AM +0900, Michael Paquier wrote: >> >> >> Strange. I can confirm that I have received the patch as attached, but >> it is not on the archives. > > It must have been stripped by our email

Re: [HACKERS] Speedup twophase transactions

On Sun, Dec 18, 2016 at 07:41:50AM +0900, Michael Paquier wrote: > On Sun, Dec 18, 2016 at 6:42 AM, Bruce Momjian wrote: > > Uh, did you mean to attached patch here? > > Strange. I can confirm that I have received the patch as attached, but > it is not on the archives. It must

Re: [HACKERS] Speedup twophase transactions

On Sun, Dec 18, 2016 at 6:42 AM, Bruce Momjian wrote: > Uh, did you mean to attached patch here? Strange. I can confirm that I have received the patch as attached, but it is not on the archives. -- Michael diff --git a/src/backend/access/transam/twophase.c

Re: [HACKERS] Speedup twophase transactions

On Fri, Dec 16, 2016 at 02:00:46PM +0300, Stas Kelvich wrote: > > > On 27 Sep 2016, at 03:30, Michael Paquier wrote: > > > > OK. I am marking this patch as returned with feedback then. Looking > > forward to seeing the next investigations.. At least this review has >

Re: [HACKERS] Speedup twophase transactions

On 27 Sep 2016, at 03:30, Michael Paquier wrote:OK. I am marking this patch as returned with feedback then. Lookingforward to seeing the next investigations.. At least this review hastaught us one thing or two.So, here is brand new implementation of the same thing.Now

Re: [HACKERS] Speedup twophase transactions

On Thu, Sep 22, 2016 at 12:30 AM, Stas Kelvich wrote: > I’m not giving up yet, i’ll write them) I still have in mind several other > patches to 2pc handling in > postgres during this release cycle — logical decoding and partitioned hash > instead of > TwoPhaseState

Re: [HACKERS] Speedup twophase transactions

> On 21 Sep 2016, at 10:32, Michael Paquier wrote: > > On Tue, Sep 20, 2016 at 11:13 PM, Stas Kelvich > wrote: >> >> Putting that before actual WAL replay is just following historical order of >> events. >> Prepared files are pieces of WAL

Re: [HACKERS] Speedup twophase transactions

On Tue, Sep 20, 2016 at 11:13 PM, Stas Kelvich wrote: >> On 20 Sep 2016, at 09:40, Michael Paquier wrote: >> + StartupCLOG(); >> + StartupSUBTRANS(checkPoint.oldestActiveXid); >> + RecoverPreparedFromFiles(); >> [...] >>/* >>

Re: [HACKERS] Speedup twophase transactions

> > On 20 Sep 2016, at 09:40, Michael Paquier wrote: > Thanks for digging into this. > + StartupCLOG(); > + StartupSUBTRANS(checkPoint.oldestActiveXid); > + RecoverPreparedFromFiles(); > [...] >/* > -* Startup commit log and subtrans

Re: [HACKERS] Speedup twophase transactions

On Sat, Sep 17, 2016 at 2:45 AM, Stas Kelvich wrote: > Looking through old version i’ve noted few things that were bothering me: > > * In case of crash replay PREPARE redo accesses SUBTRANS, but > StartupSUBTRANS() isn’t called yet > in StartupXLOG(). > * Several

Re: [HACKERS] Speedup twophase transactions

> On 07 Sep 2016, at 11:07, Stas Kelvich wrote: > >> On 07 Sep 2016, at 03:09, Michael Paquier wrote: >> On 06 Sep 2016, at 12:03, Michael Paquier wrote: On Tue, Sep 6, 2016 at 5:58 PM, Stas

Re: [HACKERS] Speedup twophase transactions

> On 07 Sep 2016, at 03:09, Michael Paquier wrote: > >>> On 06 Sep 2016, at 12:03, Michael Paquier wrote: >>> >>> On Tue, Sep 6, 2016 at 5:58 PM, Stas Kelvich >>> wrote: Oh, I was preparing new version of

Re: [HACKERS] Speedup twophase transactions

>> On 06 Sep 2016, at 12:03, Michael Paquier wrote: >> >> On Tue, Sep 6, 2016 at 5:58 PM, Stas Kelvich >> wrote: >>> Oh, I was preparing new version of patch, after fresh look on it. Probably, >>> I should >>> said that in this topic. I’ve

Re: [HACKERS] Speedup twophase transactions

> On 06 Sep 2016, at 12:09, Simon Riggs wrote: > > On 6 September 2016 at 09:58, Stas Kelvich wrote: >> >> I'll check it against my failure scenario with subtransactions and post >> results or updated patch here. > > Make sure tests are added

Re: [HACKERS] Speedup twophase transactions

On 6 September 2016 at 09:58, Stas Kelvich wrote: >> On 06 Sep 2016, at 04:41, Michael Paquier wrote: >> >> On Sat, Sep 3, 2016 at 10:26 PM, Michael Paquier >> wrote: >>> On Fri, Sep 2, 2016 at 5:06 AM, Simon Riggs

Re: [HACKERS] Speedup twophase transactions

On Tue, Sep 6, 2016 at 5:58 PM, Stas Kelvich wrote: >> On 06 Sep 2016, at 04:41, Michael Paquier wrote: >> This overlaps with TwoPhaseGetGXact but I'd rather keep both things >> separated: it does not seem worth complicating the existing

Re: [HACKERS] Speedup twophase transactions

> On 06 Sep 2016, at 04:41, Michael Paquier wrote: > > On Sat, Sep 3, 2016 at 10:26 PM, Michael Paquier > wrote: >> On Fri, Sep 2, 2016 at 5:06 AM, Simon Riggs wrote: >>> On 13 April 2016 at 15:31, Stas Kelvich

Re: [HACKERS] Speedup twophase transactions

On 6 September 2016 at 02:41, Michael Paquier wrote: > After review the result is attached. Perhaps a committer could get a look at > it? Yes, will do, but it will be a few more days yet. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] Speedup twophase transactions

On Sat, Sep 3, 2016 at 10:26 PM, Michael Paquier wrote: > On Fri, Sep 2, 2016 at 5:06 AM, Simon Riggs wrote: >> On 13 April 2016 at 15:31, Stas Kelvich wrote: >> >>> Fixed patch attached. There already was

Re: [HACKERS] Speedup twophase transactions

On Fri, Sep 2, 2016 at 5:06 AM, Simon Riggs wrote: > On 13 April 2016 at 15:31, Stas Kelvich wrote: > >> Fixed patch attached. There already was infrastructure to skip currently >> held locks during replay of standby_redo() and I’ve extended that

Re: [HACKERS] Speedup twophase transactions

On 13 April 2016 at 15:31, Stas Kelvich wrote: > Fixed patch attached. There already was infrastructure to skip currently > held locks during replay of standby_redo() and I’ve extended that with check > for > prepared xids. Please confirm that everything still works

Re: [HACKERS] Speedup twophase transactions

On Fri, May 20, 2016 at 12:46 PM, Alvaro Herrera wrote: > Jesper Pedersen wrote: >> Discussed with Noah off-list I think we should revisit this for 9.6 due to >> the async replica lag as shown in [1]. The performance improvement for the >> master node is shown in [2]. >

Re: [HACKERS] Speedup twophase transactions

Jesper Pedersen wrote: > Discussed with Noah off-list I think we should revisit this for 9.6 due to > the async replica lag as shown in [1]. The performance improvement for the > master node is shown in [2]. I gave a very quick look and it seems to me far more invasive than we would normally

Re: [HACKERS] Speedup twophase transactions

Hi, On 04/13/2016 10:31 AM, Stas Kelvich wrote: On 13 Apr 2016, at 01:04, Michael Paquier wrote: That's too late for 9.6 unfortunately, don't forget to add that in the next CF! Fixed patch attached. There already was infrastructure to skip currently held locks

Re: [HACKERS] Speedup twophase transactions

> On 13 Apr 2016, at 01:04, Michael Paquier wrote: > > On Wed, Apr 13, 2016 at 1:53 AM, Stas Kelvich > wrote: >>> On 12 Apr 2016, at 15:47, Michael Paquier wrote: >>> >>> It looks to be the case... The PREPARE

Re: [HACKERS] Speedup twophase transactions

On Wed, Apr 13, 2016 at 1:53 AM, Stas Kelvich wrote: >> On 12 Apr 2016, at 15:47, Michael Paquier wrote: >> >> It looks to be the case... The PREPARE phase replayed after the >> standby is restarted in recovery creates a series of exclusive

Re: [HACKERS] Speedup twophase transactions

> On 12 Apr 2016, at 15:47, Michael Paquier wrote: > > On Mon, Apr 11, 2016 at 7:16 PM, Stas Kelvich wrote: >> Michael, it looks like that you are the only one person who can reproduce >> that bug. I’ve tried on bunch of OS’s and didn’t observe that behaviour, >>

Re: [HACKERS] Speedup twophase transactions

On Mon, Apr 11, 2016 at 7:16 PM, Stas Kelvich wrote: > Michael, it looks like that you are the only one person who can reproduce > that bug. I’ve tried on bunch of OS’s and didn’t observe that behaviour, also > looking at your backtraces I can’t get who is holding this lock (and all of > that

Re: [HACKERS] Speedup twophase transactions

> On 08 Apr 2016, at 16:09, Stas Kelvich wrote: > > Tried on linux and os x 10.11 and 10.4. > > Still can’t reproduce, but have played around with your backtrace. > > I see in xlodump on slave following sequence of records: > > rmgr: Storage len (rec/tot):

Re: [HACKERS] Speedup twophase transactions

On Fri, Apr 08, 2016 at 02:57:00PM -0400, Jesper Pedersen wrote: > On 04/08/2016 02:37 PM, Robert Haas wrote: > >On Fri, Apr 8, 2016 at 8:49 AM, Jesper Pedersen > >wrote: > >>Should we create an entry for the open item list [0] for this, due to the > >>replication lag

Re: [HACKERS] Speedup twophase transactions

> On 08 Apr 2016, at 21:42, Robert Haas wrote: > > On Tue, Jan 26, 2016 at 7:43 AM, Stas Kelvich > wrote: >> Hi, >> >> Thanks for reviews and commit! > > I apologize for being clueless here, but was this patch committed? > It's still marked

Re: [HACKERS] Speedup twophase transactions

On 04/08/2016 02:37 PM, Robert Haas wrote: On Fri, Apr 8, 2016 at 8:49 AM, Jesper Pedersen wrote: On 04/07/2016 02:29 AM, Michael Paquier wrote: So recovery is conflicting here. My guess is that this patch is missing some lock cleanup. With the test case attached

Re: [HACKERS] Speedup twophase transactions

On 04/08/2016 02:42 PM, Robert Haas wrote: On Tue, Jan 26, 2016 at 7:43 AM, Stas Kelvich wrote: Hi, Thanks for reviews and commit! I apologize for being clueless here, but was this patch committed? It's still marked as "Needs Review" in the CommitFest application.

Re: [HACKERS] Speedup twophase transactions

On Tue, Jan 26, 2016 at 7:43 AM, Stas Kelvich wrote: > Hi, > > Thanks for reviews and commit! I apologize for being clueless here, but was this patch committed? It's still marked as "Needs Review" in the CommitFest application. -- Robert Haas EnterpriseDB:

  1   2   >