Re: Add last commit LSN to pg_last_committed_xact()

2022-08-02 Thread Jacob Champion
This entry has been waiting on author input for a while (our current threshold is roughly two weeks), so I've marked it Returned with Feedback. Once you think the patchset is ready for review again, you (or any interested party) can resurrect the patch entry by visiting https://commitfest.pos

Re: Add last commit LSN to pg_last_committed_xact()

2022-04-07 Thread Michael Paquier
On Sat, Jan 29, 2022 at 02:51:32PM -0500, James Coleman wrote: > Oh, thanks. I'd seen some discussion previously on the list about > clang not supporting it, but that seems to have been incorrect. Also I > didn't know about that compiler site -- that's really neat. > > Here's an updated patch seri

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-29 Thread James Coleman
On Fri, Jan 28, 2022 at 7:47 PM Andres Freund wrote: > > On 2022-01-28 16:36:32 -0800, Andres Freund wrote: > > On 2022-01-28 18:43:57 -0500, James Coleman wrote: > > > Alternatively I see pg_attribute_aligned, but that's not defined > > > (AFAICT) on clang, for example, so I'm not sure that'd be

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-28 Thread Andres Freund
On 2022-01-28 16:36:32 -0800, Andres Freund wrote: > On 2022-01-28 18:43:57 -0500, James Coleman wrote: > > Alternatively I see pg_attribute_aligned, but that's not defined > > (AFAICT) on clang, for example, so I'm not sure that'd be acceptable? > > clang should have it (it defines __GNUC__). The

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-28 Thread Andres Freund
Hi, On 2022-01-28 18:43:57 -0500, James Coleman wrote: > Alternatively I see pg_attribute_aligned, but that's not defined > (AFAICT) on clang, for example, so I'm not sure that'd be acceptable? clang should have it (it defines __GNUC__). The problem would be msvc, I think. Not sure if there's a w

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-28 Thread James Coleman
On Thu, Jan 20, 2022 at 8:15 AM James Coleman wrote: > > On Wed, Jan 19, 2022 at 10:12 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-01-19 21:23:12 -0500, James Coleman wrote: > > > { oid => '3537', descr => 'get identification of SQL object', > > > diff --git a/src/include/storage/proc.h

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-20 Thread James Coleman
On Wed, Jan 19, 2022 at 10:12 PM Andres Freund wrote: > > Hi, > > On 2022-01-19 21:23:12 -0500, James Coleman wrote: > > { oid => '3537', descr => 'get identification of SQL object', > > diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h > > index a5f9e9..2a026b0844 100644 >

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-19 Thread Andres Freund
Hi, On 2022-01-19 21:23:12 -0500, James Coleman wrote: > { oid => '3537', descr => 'get identification of SQL object', > diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h > index a5f9e9..2a026b0844 100644 > --- a/src/include/storage/proc.h > +++ b/src/include/storage/proc.h

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-19 Thread James Coleman
On Tue, Jan 18, 2022 at 9:19 PM Andres Freund wrote: > > > + LWLockAcquire(ProcArrayLock, LW_SHARED); > > + lsn = ShmemVariableCache->finishedProcsLastCommitLSN; > > + for (index = 0; index < ProcGlobal->allProcCount; index++) > > + { > > + XLogRecPtr procLSN = > > Pro

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Andres Freund
Hi, On 2022-01-18 20:58:01 -0500, James Coleman wrote: > Is something roughly like the attached what you'd envisioned? Roughly, yea. > I think we need a shared ProcArrayLock to read the array, correct? You could perhaps get away without it, but it'd come at the price of needing to look at all

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 4:32 PM Andres Freund wrote: > > I wonder if a very different approach could make sense here. Presumably this > wouldn't need to be queried at a very high frequency, right? If so, what about > storing the latest commit LSN for each backend in PGPROC? That could be > maintai

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 8:05 PM Andres Freund wrote: > > Hi, > > On 2022-01-18 18:31:42 -0500, James Coleman wrote: > > One other question on this: if we went with this would you expect a > > new function to parallel pg_last_committed_xact()? > > I don't think I have an opinion the user interface

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Andres Freund
Hi, On 2022-01-18 18:31:42 -0500, James Coleman wrote: > One other question on this: if we went with this would you expect a > new function to parallel pg_last_committed_xact()? I don't think I have an opinion the user interface aspect. > Or allow the xid and lsn in the return of pg_last_commit

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 4:32 PM Andres Freund wrote: > I wonder if a very different approach could make sense here. Presumably this > wouldn't need to be queried at a very high frequency, right? If so, what about > storing the latest commit LSN for each backend in PGPROC? That could be > maintaine

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Andres Freund
On 2022-01-18 16:40:25 -0500, James Coleman wrote: > If I read between the lines I imagine you'd see even e.g. every 2s as > not that big of a deal here, right? Right. Even every 0.2s wouldn't be a problem.

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 4:32 PM Andres Freund wrote: > > Hi, > > On 2022-01-17 18:34:16 -0300, Alvaro Herrera wrote: > > Maybe it would work to have a single LSN in shared memory, as an atomic > > variable, which uses monotonic advance[1] to be updated. > > That could be a reasonable approach. > >

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Andres Freund
Hi, On 2022-01-17 18:34:16 -0300, Alvaro Herrera wrote: > Maybe it would work to have a single LSN in shared memory, as an atomic > variable, which uses monotonic advance[1] to be updated. That could be a reasonable approach. > Whether this is updated or not would depend on a new GUC, maybe > t

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 1:52 PM Alvaro Herrera wrote: > > On 2022-Jan-18, James Coleman wrote: > > > Reading the code it seems the only usage (besides > > the boolean activation status also stored there) is in > > TransactionIdGetCommitTsData, and the only consumers of that in core > > appear to b

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Alvaro Herrera
On 2022-Jan-18, James Coleman wrote: > Reading the code it seems the only usage (besides > the boolean activation status also stored there) is in > TransactionIdGetCommitTsData, and the only consumers of that in core > appear to be the SQL callable functions to get the latest commit info. > It is

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 12:50 PM Alvaro Herrera wrote: > > On 2022-Jan-17, James Coleman wrote: > > > I'd be happy to make it a separate GUC, though it seems adding an > > additional atomic access is worse (assuming we can convince ourselves > > putting this into the commit timestamps infrastructu

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Alvaro Herrera
On 2022-Jan-17, James Coleman wrote: > I'd be happy to make it a separate GUC, though it seems adding an > additional atomic access is worse (assuming we can convince ourselves > putting this into the commit timestamps infrastructure is acceptable) > given here we're already under a lock. I was t

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Robert Haas
On Tue, Jan 18, 2022 at 9:47 AM James Coleman wrote: > > Well, if you're maintaining an SLRU, you do kind of need to know where > > the leading and lagging ends are. > > As far as I can tell the data in commitTsShared is used purely as an > optimization for the path looking up the timestamp for an

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 9:25 AM Robert Haas wrote: > > On Mon, Jan 17, 2022 at 8:39 PM James Coleman wrote: > > I wondered about that, but commit_ts already does more than commit > > timestamps by recording the xid of the last commit. > > Well, if you're maintaining an SLRU, you do kind of need t

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Robert Haas
On Mon, Jan 17, 2022 at 8:39 PM James Coleman wrote: > I wondered about that, but commit_ts already does more than commit > timestamps by recording the xid of the last commit. Well, if you're maintaining an SLRU, you do kind of need to know where the leading and lagging ends are. > For that matt

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread Alvaro Herrera
On 2022-Jan-17, Robert Haas wrote: > On Mon, Jan 17, 2022 at 4:34 PM Alvaro Herrera > wrote: > > Maybe it would work to have a single LSN in shared memory, as an atomic > > variable, which uses monotonic advance[1] to be updated. Whether this is > > updated or not would depend on a new GUC, ma

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread James Coleman
On Mon, Jan 17, 2022 at 4:34 PM Alvaro Herrera wrote: > > On 2022-Jan-14, James Coleman wrote: > > > The logical slot can't flush past the > > last commit, so even if there's 100s of megabytes of unflushed WAL on > > the slot there may be zero lag (in terms of what's possible to > > process). > >

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread James Coleman
On Mon, Jan 17, 2022 at 4:20 PM Robert Haas wrote: > > On Fri, Jan 14, 2022 at 7:42 PM James Coleman wrote: > > I've attached a simple patch (sans tests and documentation) to get > > feedback early. After poking around this afternoon it seemed to me > > that the simplest approach was to hook into

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 4:34 PM Alvaro Herrera wrote: > On 2022-Jan-14, James Coleman wrote: > > The logical slot can't flush past the > > last commit, so even if there's 100s of megabytes of unflushed WAL on > > the slot there may be zero lag (in terms of what's possible to > > process). > > > >

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread Alvaro Herrera
On 2022-Jan-14, James Coleman wrote: > The logical slot can't flush past the > last commit, so even if there's 100s of megabytes of unflushed WAL on > the slot there may be zero lag (in terms of what's possible to > process). > > I've attached a simple patch (sans tests and documentation) to get >

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread Robert Haas
On Fri, Jan 14, 2022 at 7:42 PM James Coleman wrote: > I've attached a simple patch (sans tests and documentation) to get > feedback early. After poking around this afternoon it seemed to me > that the simplest approach was to hook into the commit timestamps > infrastructure and store the commit's

Add last commit LSN to pg_last_committed_xact()

2022-01-14 Thread James Coleman
I'd recently been thinking about monitoring how many bytes behind a logical slot was and realized that's not really possible to compute currently. That's easy enough with a physical slot because we can get the current WAL LSN easily enough and the slot exposes the current LSN positions of the slot.