Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-09-17 Thread Alexander Lakhin
17.09.2024 10:47, Alexander Korotkov wrote: Yes, now I did reproduce. I got that the problem could be that insert LSN is not yet written at primary, thus wait_for_catchup doesn't wait for it. I've workarounded that using pg_switch_wal(). The revised patchset is attached. Thank you for the re

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-09-17 Thread Alexander Korotkov
On Tue, Sep 17, 2024 at 9:00 AM Alexander Lakhin wrote: > 16.09.2024 21:55, Alexander Korotkov wrote: > > Please find two patches attached. The first one does minor cleanup > > including misuse of words you've pointed. The second one adds missing > > wait_for_catchup(). That should fix the test

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-09-16 Thread Alexander Lakhin
Hi Alexander, 16.09.2024 21:55, Alexander Korotkov wrote: Please find two patches attached. The first one does minor cleanup including misuse of words you've pointed. The second one adds missing wait_for_catchup(). That should fix the test failure you've spotted. Please, check if it fixes an

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-09-16 Thread Alexander Korotkov
Hi, Alexander! On Fri, Sep 13, 2024 at 3:00 PM Alexander Lakhin wrote: > > 10.08.2024 20:18, Alexander Korotkov wrote: > > On Sat, Aug 10, 2024 at 7:33 PM Alexander Korotkov > > wrote: > >> On Tue, Aug 6, 2024 at 5:17 AM Alexander Korotkov > >> wrote: > >> ... > >> Here is a revised version o

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-09-13 Thread Alexander Lakhin
Hi Alexander, 10.08.2024 20:18, Alexander Korotkov wrote: On Sat, Aug 10, 2024 at 7:33 PM Alexander Korotkov wrote: On Tue, Aug 6, 2024 at 5:17 AM Alexander Korotkov wrote: ... Here is a revised version of the patchset. I've fixed some typos, identation, etc. I'm going to push this once it

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-12 Thread Alexander Korotkov
On Sat, Aug 10, 2024 at 6:58 PM Alexander Korotkov wrote: > On Tue, Aug 6, 2024 at 8:36 AM Michael Paquier wrote: > > On Tue, Aug 06, 2024 at 05:17:10AM +0300, Alexander Korotkov wrote: > > > The 0001 patch is intended to improve this situation. Actually, it's > > > not right to just put Recover

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-10 Thread Alexander Korotkov
On Sat, Aug 10, 2024 at 7:33 PM Alexander Korotkov wrote: > On Tue, Aug 6, 2024 at 5:17 AM Alexander Korotkov > wrote: > > On Sat, Aug 3, 2024 at 6:07 AM Alexander Korotkov > > wrote: > > > On Sat, Aug 3, 2024 at 3:45 AM Kevin Hale Boyes wrote: > > > > In the for loop in WaitForLSNReplay, sho

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-10 Thread Alexander Korotkov
On Tue, Aug 6, 2024 at 5:17 AM Alexander Korotkov wrote: > On Sat, Aug 3, 2024 at 6:07 AM Alexander Korotkov > wrote: > > On Sat, Aug 3, 2024 at 3:45 AM Kevin Hale Boyes wrote: > > > In the for loop in WaitForLSNReplay, shouldn't the check for in-recovery > > > be moved up above the call to Ge

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-10 Thread Alexander Korotkov
On Tue, Aug 6, 2024 at 8:36 AM Michael Paquier wrote: > On Tue, Aug 06, 2024 at 05:17:10AM +0300, Alexander Korotkov wrote: > > The 0001 patch is intended to improve this situation. Actually, it's > > not right to just put RecoveryInProgress() after > > GetXLogReplayRecPtr(), because more wal cou

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-06 Thread Alexander Korotkov
On Tue, Aug 6, 2024 at 11:18 AM Alexander Korotkov wrote: > On Tue, Aug 6, 2024 at 8:36 AM Michael Paquier wrote: > > On Tue, Aug 06, 2024 at 05:17:10AM +0300, Alexander Korotkov wrote: > > > The 0001 patch is intended to improve this situation. Actually, it's > > > not right to just put Recover

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-06 Thread Alexander Korotkov
On Tue, Aug 6, 2024 at 8:36 AM Michael Paquier wrote: > On Tue, Aug 06, 2024 at 05:17:10AM +0300, Alexander Korotkov wrote: > > The 0001 patch is intended to improve this situation. Actually, it's > > not right to just put RecoveryInProgress() after > > GetXLogReplayRecPtr(), because more wal cou

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-05 Thread Michael Paquier
On Tue, Aug 06, 2024 at 05:17:10AM +0300, Alexander Korotkov wrote: > The 0001 patch is intended to improve this situation. Actually, it's > not right to just put RecoveryInProgress() after > GetXLogReplayRecPtr(), because more wal could be replayed between > these calls. Instead we need to reche

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-05 Thread Alexander Korotkov
On Sat, Aug 3, 2024 at 6:07 AM Alexander Korotkov wrote: > On Sat, Aug 3, 2024 at 3:45 AM Kevin Hale Boyes wrote: > > In the for loop in WaitForLSNReplay, shouldn't the check for in-recovery be > > moved up above the call to GetXLogReplayRecPtr? > > If we get promoted while waiting for the timeo

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-02 Thread Alexander Korotkov
On Sat, Aug 3, 2024 at 3:45 AM Kevin Hale Boyes wrote: > I noticed the commit and had a question and a comment. > There is a small problem in func.sgml in the sentence "After that the changes > made of primary". Should be "on primary". Thank you for spotting this. Will fix. > In the for loop i

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-08-02 Thread Kevin Hale Boyes
I noticed the commit and had a question and a comment. There is a small problem in func.sgml in the sentence "After that the changes made of primary". Should be "on primary". In the for loop in WaitForLSNReplay, shouldn't the check for in-recovery be moved up above the call to GetXLogReplayRecPtr

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-07-31 Thread Alexander Korotkov
On Mon, Jul 15, 2024 at 2:02 PM Alexander Korotkov wrote: > On Mon, Jul 15, 2024 at 4:24 AM Alexander Korotkov > wrote: > > Thanks to Kyotaro for the review. And thanks to Ivan for the patch > > revision. I made another revision of the patch. > > I've noticed failures on cfbot. The attached r

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-07-15 Thread Alexander Korotkov
On Mon, Jul 15, 2024 at 4:24 AM Alexander Korotkov wrote: > Thanks to Kyotaro for the review. And thanks to Ivan for the patch > revision. I made another revision of the patch. I've noticed failures on cfbot. The attached revision addressed docs build failure. Also it adds some "quits" for ba

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-07-14 Thread Alexander Korotkov
Thanks to Kyotaro for the review. And thanks to Ivan for the patch revision. I made another revision of the patch. On Tue, Jul 9, 2024 at 12:51 PM Kartyshov Ivan wrote: > Thank you for your interest in the patch. > > On 2024-06-20 11:30, Kyotaro Horiguchi wrote: > > Hi, I looked through the pat

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-07-09 Thread Kartyshov Ivan
Thank you for your interest in the patch. On 2024-06-20 11:30, Kyotaro Horiguchi wrote: Hi, I looked through the patch and have some comments. == L68: +Recovery Procedures It looks somewhat confusing and appears as if the section is intended to explain how to perform recovery. Since t

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-06-20 Thread Kyotaro Horiguchi
Hi, I looked through the patch and have some comments. == L68: +Recovery Procedures It looks somewhat confusing and appears as if the section is intended to explain how to perform recovery. Since this is the first built-in procedure, I'm not sure how should this section be written. Howev

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-06-19 Thread Alexander Korotkov
On Fri, Jun 14, 2024 at 3:46 PM Alexander Korotkov wrote: > On Wed, Jun 12, 2024 at 11:36 AM Kartyshov Ivan > wrote: > > > > Hi, Alexander, Here, I made some improvements according to your > > discussion with Heikki. > > > > On 2024-04-11 18:09, Alexander Korotkov wrote: > > > On Thu, Apr 11, 202

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-06-14 Thread Alexander Korotkov
Hi, Ivan! On Wed, Jun 12, 2024 at 11:36 AM Kartyshov Ivan wrote: > > Hi, Alexander, Here, I made some improvements according to your > discussion with Heikki. > > On 2024-04-11 18:09, Alexander Korotkov wrote: > > On Thu, Apr 11, 2024 at 1:46 AM Heikki Linnakangas > > wrote: > >> In a nutshell,

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-06-12 Thread Kartyshov Ivan
Hi, Alexander, Here, I made some improvements according to your discussion with Heikki. On 2024-04-11 18:09, Alexander Korotkov wrote: On Thu, Apr 11, 2024 at 1:46 AM Heikki Linnakangas wrote: In a nutshell, it's possible for the loop in WaitForLSN to exit without cleaning up the process from

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-11 Thread Heikki Linnakangas
On 11/04/2024 18:09, Alexander Korotkov wrote: On Thu, Apr 11, 2024 at 1:46 AM Heikki Linnakangas wrote: On 07/04/2024 00:52, Alexander Korotkov wrote: * At first, we check that pg_wal_replay_wait() is called in a non-atomic * context. That is, a procedure call isn't wrapped i

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-11 Thread Alexander Korotkov
Hi, Heikki! Thank you for your interest in the subject. On Thu, Apr 11, 2024 at 1:46 AM Heikki Linnakangas wrote: > On 07/04/2024 00:52, Alexander Korotkov wrote: > > On Fri, Apr 5, 2024 at 9:15 PM Alvaro Herrera > > wrote: > >> I'm still concerned that WaitLSNCleanup is only called in ProcKil

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-10 Thread Heikki Linnakangas
On 07/04/2024 00:52, Alexander Korotkov wrote: On Fri, Apr 5, 2024 at 9:15 PM Alvaro Herrera wrote: I'm still concerned that WaitLSNCleanup is only called in ProcKill. Does this mean that if a process throws an error while waiting, it'll not get cleaned up until it exits? Maybe this is not a b

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-10 Thread Kartyshov Ivan
I did some experiments over synchronous replications and got that cascade replication can`t be synchronous. And  pg_wal_replay_wait() allows us to read your writes consistency on cascade replication. Beyond that, I added more tests on multi-standby replication and cascade replications. -- Ivan Ka

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-06 Thread Alexander Korotkov
Hi, Alvaro! Thank you for your care on this matter. On Fri, Apr 5, 2024 at 9:15 PM Alvaro Herrera wrote: > BTW I noticed that > https://coverage.postgresql.org/src/backend/commands/waitlsn.c.gcov.html > says that lsn_cmp is not covered by the tests. This probably indicates > that the tests are

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-05 Thread Alvaro Herrera
Hello, BTW I noticed that https://coverage.postgresql.org/src/backend/commands/waitlsn.c.gcov.html says that lsn_cmp is not covered by the tests. This probably indicates that the tests are a little too light, but I'm not sure how much extra effort we want to spend. I'm still concerned that Wait

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alexander Korotkov
On Wed, Apr 3, 2024 at 10:04 PM Pavel Borisov wrote: > On Wed, 3 Apr 2024 at 22:18, Alexander Korotkov wrote: >> >> On Wed, Apr 3, 2024 at 7:55 PM Alvaro Herrera >> wrote: >> > >> > On 2024-Apr-03, Alexander Korotkov wrote: >> > >> > > Regarding the shmem data structure for LSN waiters. I didn

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Pavel Borisov
Hi, Alexander! On Wed, 3 Apr 2024 at 22:18, Alexander Korotkov wrote: > On Wed, Apr 3, 2024 at 7:55 PM Alvaro Herrera > wrote: > > > > On 2024-Apr-03, Alexander Korotkov wrote: > > > > > Regarding the shmem data structure for LSN waiters. I didn't pick > > > LWLock or ConditionVariable, becaus

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alexander Korotkov
On Wed, Apr 3, 2024 at 7:55 PM Alvaro Herrera wrote: > > On 2024-Apr-03, Alexander Korotkov wrote: > > > Regarding the shmem data structure for LSN waiters. I didn't pick > > LWLock or ConditionVariable, because I needed the ability to wake up > > only those waiters whose LSN is already replayed.

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alvaro Herrera
Hello Alexander, On 2024-Apr-03, Alexander Korotkov wrote: > Regarding the shmem data structure for LSN waiters. I didn't pick > LWLock or ConditionVariable, because I needed the ability to wake up > only those waiters whose LSN is already replayed. In my experience > waking up a process is way

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alexander Korotkov
Hi, Alvaro! Thank you for your feedback. On Wed, Apr 3, 2024 at 9:58 AM Alvaro Herrera wrote: > Hello, I noticed that commit 06c418e163e9 uses waitLSN->mutex (a > spinlock) to protect the contents of waitLSN -- and it's used to walk an > arbitrary long list of processes waiting ... and also, an

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Andy Fan
> I also can confirm that a lot of users would be very happy to have > "read your writes consistency" and ready to do something to achieve > this at an application level. However, they typically don't know what > exactly they need. > > So, blogging about pg_wal_replay_wait() and spreading words

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Daniel Gustafsson
Buildfarm animal mamba (NetBSD 10.0 on macppc) started failing on this with what seems like a bogus compiler warning: waitlsn.c: In function 'WaitForLSN': waitlsn.c:275:24: error: 'endtime' may be used uninitialized in this function [-Werror=maybe-uninitialized] 275 |delay_ms = (endtime - G

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Alvaro Herrera
Hello, I noticed that commit 06c418e163e9 uses waitLSN->mutex (a spinlock) to protect the contents of waitLSN -- and it's used to walk an arbitrary long list of processes waiting ... and also, an arbitrary number of processes could be calling this code. I think using a spinlock for this is unwise,

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Kartyshov Ivan
On 2024-04-02 13:15, Bharath Rupireddy wrote: On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan wrote: 8 years, we tried to add this feature, and now we suggest the best way for this feature is to commit the minimal version first. Just curious, do you or anyone else have an immediate use for thi

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Alexander Korotkov
On Tue, Apr 2, 2024 at 2:47 PM Andy Fan wrote: > Bharath Rupireddy writes: > > > On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan > > wrote: > >> > >> 8 years, we tried to add this feature, and now we suggest the best way > >> for this feature is to commit the minimal version first. > > > > Just cu

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Alexander Korotkov
On Tue, Apr 2, 2024 at 1:11 PM Kartyshov Ivan wrote: > On 2024-04-02 11:14, Andy Fan wrote: > > If so, when users call pg_wal_replay_wait, they can get informed when > > the wal is replaied to the target_lsn, but they can't know how long > > time > > it waits unless they check it in application si

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Andy Fan
Bharath Rupireddy writes: > On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan > wrote: >> >> 8 years, we tried to add this feature, and now we suggest the best way >> for this feature is to commit the minimal version first. > > Just curious, do you or anyone else have an immediate use for this > f

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Bharath Rupireddy
On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan wrote: > > 8 years, we tried to add this feature, and now we suggest the best way > for this feature is to commit the minimal version first. Just curious, do you or anyone else have an immediate use for this function? If yes, how are they achieving re

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Kartyshov Ivan
On 2024-04-02 11:14, Andy Fan wrote: If so, when users call pg_wal_replay_wait, they can get informed when the wal is replaied to the target_lsn, but they can't know how long time it waits unless they check it in application side, I think such information will be useful for monitor purpose some

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Andy Fan
Hi Alexander! >> + >> +pg_wal_replay_wait ( >> + target_lsn pg_lsn, >> + timeout bigint >> DEFAULT 0) >> +void >> + >> >> Should we return the millseconds of waiting time? I think this >> information may be useful for customer if they want to kn

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Alexander Korotkov
Hi, Andy! On Tue, Apr 2, 2024 at 6:29 AM Andy Fan wrote: > > Did you forget to attach the new patch? > > > > Yes, here it is. > > > > [4. text/x-diff; > > v17-0001-Implement-pg_wal_replay_wait-stored-procedure.patch]... > > + > +pg_wal_replay_wait ( > + target_lsn pg_ls

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-01 Thread Andy Fan
Alexander Korotkov writes: Hello, > > Did you forget to attach the new patch? > > Yes, here it is. > > -- > Regards, > Alexander Korotkov > > [4. text/x-diff; > v17-0001-Implement-pg_wal_replay_wait-stored-procedure.patch]... + +pg_wal_replay_wait ( + target

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-01 Thread Alexander Korotkov
On Mon, Apr 1, 2024 at 5:25 AM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > On Mon, Apr 1, 2024 at 5:54 AM Alexander Korotkov > wrote: > > > > > 9. To me the following query blocks even though I didn't mention > timeout. > > > CALL pg_wal_replay_wait('0/fff'); > > > >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-31 Thread Bharath Rupireddy
On Mon, Apr 1, 2024 at 5:54 AM Alexander Korotkov wrote: > > > 9. To me the following query blocks even though I didn't mention timeout. > > CALL pg_wal_replay_wait('0/fff'); > > If your primary server is freshly initialized, you need to do quite > data modifications to reach this LSN. Right,

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-31 Thread Alexander Korotkov
Hi Bharath, Thank you for your feedback. On Sun, Mar 31, 2024 at 8:44 AM Bharath Rupireddy wrote: > On Sun, Mar 31, 2024 at 7:41 AM Alexander Korotkov > wrote: > Thanks for the patch. I have a few comments on the v16 patch. > > 1. Is there any specific reason for pg_wal_replay_wait() being a >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Bharath Rupireddy
On Sun, Mar 31, 2024 at 7:41 AM Alexander Korotkov wrote: > > Hi! Thanks for the patch. I have a few comments on the v16 patch. 1. Is there any specific reason for pg_wal_replay_wait() being a procedure rather than a function? I haven't read the full thread, but I vaguely noticed the discussion

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Alexander Korotkov
Hi! On Sat, Mar 30, 2024 at 6:14 PM Kartyshov Ivan wrote: > > Thank you Alexander for working on patch, may be we should change some > names: > 1) test 043_wait_lsn.pl -> to 043_waitlsn.pl like waitlsn.c and > waitlsn.h I renamed that to 043_wal_replay_wait.pl to match the name of SQL procedure.

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Kartyshov Ivan
Thank you Alexander for working on patch, may be we should change some names: 1) test 043_wait_lsn.pl -> to 043_waitlsn.pl like waitlsn.c and waitlsn.h In waitlsn.c and waitlsn.h variables: 2) targret_lsn -> trgLSN like curLSN 3) lsn -> trgLSN like curLSN -- Ivan Kartyshov Postgres Professio

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Alexander Korotkov
On Fri, Mar 29, 2024 at 6:50 PM Euler Taveira wrote: > On Fri, Mar 29, 2024, at 9:44 AM, Alexander Korotkov wrote: > > This generally makes sense, but I'm not sure about this. The > milliseconds timeout was used initially but received critics in [1]. > > > Alexander, I see why you changed the pat

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Euler Taveira
On Fri, Mar 29, 2024, at 9:44 AM, Alexander Korotkov wrote: > This generally makes sense, but I'm not sure about this. The > milliseconds timeout was used initially but received critics in [1]. Alexander, I see why you changed the patch. Peter suggested to use an interval but you proposed anothe

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Pavel Borisov
Hi, hackers! On Fri, 29 Mar 2024 at 16:45, Alexander Korotkov wrote: > Hi, Euler! > > On Fri, Mar 29, 2024 at 1:38 AM Euler Taveira wrote: > > On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > > > > Fixed along with other issues spotted by Alexander Lakhin. > > > > > > [I didn't rea

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Alexander Korotkov
Hi, Euler! On Fri, Mar 29, 2024 at 1:38 AM Euler Taveira wrote: > On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > > Fixed along with other issues spotted by Alexander Lakhin. > > > [I didn't read the whole thread. I'm sorry if I missed something ...] > > You renamed the function in

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Euler Taveira
On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > Fixed along with other issues spotted by Alexander Lakhin. [I didn't read the whole thread. I'm sorry if I missed something ...] You renamed the function in a previous version but let me suggest another one: pg_wal_replay_wait. It uses

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Alexander Korotkov
On Fri, Mar 22, 2024 at 3:45 PM Kartyshov Ivan wrote: > On 2024-03-20 12:11, Alexander Korotkov wrote: > > On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan > > wrote: > >> > 4.2 With an unreasonably high future LSN, BEGIN command waits > >> > unboundedly, shouldn't we check if the specified LSN is

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Alexander Korotkov
On Thu, Mar 28, 2024 at 9:37 AM Thomas Munro wrote: > > > v12 > > Hi all, > > I didn't review the patch but one thing jumped out: I don't think it's > OK to hold a spinlock while (1) looping over an array of backends and > (2) making system calls (SetLatch()). Good catch, thank you. Fixed along

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Thomas Munro
> v12 Hi all, I didn't review the patch but one thing jumped out: I don't think it's OK to hold a spinlock while (1) looping over an array of backends and (2) making system calls (SetLatch()).

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-27 Thread Alexander Korotkov
On Thu, Mar 28, 2024 at 2:24 AM Alexander Korotkov wrote: > On Tue, Mar 26, 2024 at 4:06 PM Kartyshov Ivan > wrote: > > Thank you for your interest to the patch. > > I understand you questions, but I fully support Alexander Korotkov idea > > to commit the minimal required functionality. And then

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-27 Thread Alexander Korotkov
On Tue, Mar 26, 2024 at 4:06 PM Kartyshov Ivan wrote: > Thank you for your interest to the patch. > I understand you questions, but I fully support Alexander Korotkov idea > to commit the minimal required functionality. And then keep working on > other improvements. I did further improvements in

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Alexander Korotkov
On Sun, Mar 24, 2024 at 4:39 AM Bharath Rupireddy wrote: > I share the same concern as yours and had proposed something upthread > [1]. The idea is something like how each query takes a snapshot at the > beginning of txn/query (depending on isolation level), the same way > the standby can wait for

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Alexander Korotkov
On Fri, Mar 22, 2024 at 12:58 AM Peter Eisentraut wrote: > On 17.03.24 15:09, Alexander Korotkov wrote: > > My current attempt was to commit minimal implementation as less > > invasive as possible. A new clause for BEGIN doesn't require > > additional keywords and doesn't introduce additional sta

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Alexander Korotkov
On Fri, Mar 22, 2024 at 12:50 AM Peter Eisentraut wrote: > On 19.03.24 18:38, Kartyshov Ivan wrote: > > CALL pg_wait_lsn('0/3002AE8', 1); > > BEGIN; > > SELECT * FROM tbl; // read fresh insertions > > COMMIT; > > I'm not endorsing this or

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Kartyshov Ivan
Thank you for your interest to the patch. I understand you questions, but I fully support Alexander Korotkov idea to commit the minimal required functionality. And then keep working on other improvements. On 2024-03-24 05:39, Bharath Rupireddy wrote: On Fri, Mar 22, 2024 at 4:28 AM Peter Eisentra

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-23 Thread Bharath Rupireddy
On Fri, Mar 22, 2024 at 4:28 AM Peter Eisentraut wrote: > > I had written in [0] about my questions related to using this with > connection poolers. I don't think this was addressed at all. I haven't > seen any discussion about how to make this kind of facility usable in a > full system. You ha

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-22 Thread Kartyshov Ivan
Thank you for your feedback. On 2024-03-20 12:11, Alexander Korotkov wrote: On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > 4.2 With an unreasonably high future LSN, BEGIN command waits > unboundedly, shouldn't we check if the specified LSN is more than > pg_last_wal_receive_lsn() err

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-22 Thread Kartyshov Ivan
Thank you for your feedback. On 2024-03-20 12:11, Alexander Korotkov wrote: On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > 4.2 With an unreasonably high future LSN, BEGIN command waits > unboundedly, shouldn't we check if the specified LSN is more than > pg_last_wal_receive_lsn() err

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-22 Thread Kartyshov Ivan
On 2024-03-20 12:11, Alexander Korotkov wrote: On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > 4.2 With an unreasonably high future LSN, BEGIN command waits > unboundedly, shouldn't we check if the specified LSN is more than > pg_last_wal_receive_lsn() error out? I think limiting wai

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-21 Thread Peter Eisentraut
On 17.03.24 15:09, Alexander Korotkov wrote: My current attempt was to commit minimal implementation as less invasive as possible. A new clause for BEGIN doesn't require additional keywords and doesn't introduce additional statements. But yes, this is still a new qual. And, yes, Amit you're ri

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-21 Thread Peter Eisentraut
On 19.03.24 18:38, Kartyshov Ivan wrote: CALL pg_wait_lsn('0/3002AE8', 1); BEGIN; SELECT * FROM tbl; // read fresh insertions COMMIT; I'm not endorsing this or any other approach, but I think the timeout parameter should be of type in

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-20 Thread Alexander Korotkov
On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > > 4.2 With an unreasonably high future LSN, BEGIN command waits > > unboundedly, shouldn't we check if the specified LSN is more than > > pg_last_wal_receive_lsn() error out? I think limiting wait lsn by current received lsn would destroy t

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Kartyshov Ivan
Bharath Rupireddy, thank you for you review. But here is some points. On 2024-03-16 10:02, Bharath Rupireddy wrote: 4.1 With invalid LSN succeeds, shouldn't it error out? Or at least, add a fast path/quick exit to WaitForLSN()? BEGIN AFTER '0/0'; In postgresql '0/0' is Valid pg_lsn, but it is

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Kartyshov Ivan
Intro == The main purpose of the feature is to achieve read-your-writes-consistency, while using async replica for reads and primary for writes. In that case lsn of last modification is stored inside application. We cannot store this lsn inside database, since reads are distributed across

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Alexander Korotkov
On Tue, Mar 19, 2024 at 1:51 PM Amit Kapila wrote: > On Mon, Mar 18, 2024 at 3:24 PM Alexander Korotkov > wrote: > > > > On Mon, Mar 18, 2024 at 5:17 AM Amit Kapila wrote: > > > > 1. First, check that it was called with non-atomic context (that is, > > > > it's not called within a transaction).

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Amit Kapila
On Mon, Mar 18, 2024 at 3:24 PM Alexander Korotkov wrote: > > On Mon, Mar 18, 2024 at 5:17 AM Amit Kapila wrote: > > > 1. First, check that it was called with non-atomic context (that is, > > > it's not called within a transaction). Trigger error if called with > > > atomic context. > > > 2. Rele

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-18 Thread Alexander Korotkov
On Mon, Mar 18, 2024 at 5:17 AM Amit Kapila wrote: > On Sun, Mar 17, 2024 at 7:40 PM Alexander Korotkov wrote: > > On Sat, Mar 16, 2024 at 5:05 PM Bharath Rupireddy > > wrote: > > > On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > > > In general, it seems this patch has been stuck for a l

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-17 Thread Amit Kapila
On Sun, Mar 17, 2024 at 7:40 PM Alexander Korotkov wrote: > > On Sat, Mar 16, 2024 at 5:05 PM Bharath Rupireddy > wrote: > > On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > > In general, it seems this patch has been stuck for a long time on the > > > decision to choose an appropriate UI (

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-17 Thread Alexander Korotkov
Hi Amit, Hi Bharath, On Sat, Mar 16, 2024 at 5:05 PM Bharath Rupireddy wrote: > On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > In general, it seems this patch has been stuck for a long time on the > > decision to choose an appropriate UI (syntax), and we thought of > > moving it further

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-16 Thread Bharath Rupireddy
On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > On Fri, Mar 15, 2024 at 7:50 PM Alexander Korotkov > wrote: > > > > I went through this patch another time, and made some minor > > adjustments. Now it looks good, I'm going to push it if no > > objections. > > > > I have a question related

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-16 Thread Amit Kapila
On Fri, Mar 15, 2024 at 7:50 PM Alexander Korotkov wrote: > > I went through this patch another time, and made some minor > adjustments. Now it looks good, I'm going to push it if no > objections. > I have a question related to usability, if the regular reads (say a Select statement or reads via

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-16 Thread Bharath Rupireddy
On Sat, Mar 16, 2024 at 2:04 AM Alexander Korotkov wrote: > > > Rebase and update patch. Thanks for working on this. I took a quick look at v11 patch. Here are some comments: 1. +#include "utils/timestamp.h" +#include "executor/spi.h" +#include "utils/fmgrprotos.h" Please place executor/spi.h i

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Alexander Korotkov
On Fri, Mar 15, 2024 at 10:32 PM Kartyshov Ivan wrote: > > On 2024-03-15 22:59, Kartyshov Ivan wrote: > > On 2024-03-11 13:44, Alexander Korotkov wrote: > >> I picked the second option and left only the AFTER clause for the > >> BEGIN statement. I think this should be enough for the beginning. >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Kartyshov Ivan
On 2024-03-15 22:59, Kartyshov Ivan wrote: On 2024-03-11 13:44, Alexander Korotkov wrote: I picked the second option and left only the AFTER clause for the BEGIN statement. I think this should be enough for the beginning. Thank you for your rework on your patch, here I made some fixes: 0) aut

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Kartyshov Ivan
On 2024-03-11 13:44, Alexander Korotkov wrote: I picked the second option and left only the AFTER clause for the BEGIN statement. I think this should be enough for the beginning. Thank you for your rework on your patch, here I made some fixes: 0) autocomplete 1) less jumps 2) more description

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Alexander Korotkov
On Fri, Mar 15, 2024 at 4:20 PM Alexander Korotkov wrote: > On Mon, Mar 11, 2024 at 12:44 PM Alexander Korotkov > wrote: > > I've decided to put my hands on this patch. > > > > On Thu, Mar 7, 2024 at 2:25 PM Amit Kapila > wrote: > > > +1 for the second one not only because it avoids new words i

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Alexander Korotkov
On Mon, Mar 11, 2024 at 12:44 PM Alexander Korotkov wrote: > I've decided to put my hands on this patch. > > On Thu, Mar 7, 2024 at 2:25 PM Amit Kapila wrote: > > +1 for the second one not only because it avoids new words in grammar > > but also sounds to convey the meaning. I think you can expla

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-11 Thread Alexander Korotkov
Hi! I've decided to put my hands on this patch. On Thu, Mar 7, 2024 at 2:25 PM Amit Kapila wrote: > +1 for the second one not only because it avoids new words in grammar > but also sounds to convey the meaning. I think you can explain in docs > how this feature can be used basically how will one

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-07 Thread Amit Kapila
On Thu, Mar 7, 2024 at 5:14 PM Kartyshov Ivan wrote: > > Intro > == > The main purpose of the feature is to achieve > read-your-writes-consistency, while using async replica for reads and > primary for writes. In that case lsn of last modification is stored > inside application. We cannot

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-07 Thread Kartyshov Ivan
Intro == The main purpose of the feature is to achieve read-your-writes-consistency, while using async replica for reads and primary for writes. In that case lsn of last modification is stored inside application. We cannot store this lsn inside database, since reads are distributed across

Fwd: Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-02-01 Thread Kartyshov Ivan
Updated, rebased, fixed Ci and added documentation. We left two different solutions. Help me please to choose the best. 1) Classic (wait_classic_v6.patch) https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru == advantages: multiple wait events, separate

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-21 Thread Dilip Kumar
On Wed, Jan 17, 2024 at 1:46 PM Kartyshov Ivan wrote: > > Add some fixes and rebase. > While quickly looking into the patch, I understood the idea of what we are trying to achieve here and I feel that a useful feature. But while looking at both the patches I could not quickly differentiate betwee

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-21 Thread Peter Smith
2024-01 Commitfest. Hi, This patch has a CF status of "Needs Review" [1], but it seems there was a CFbot test failure last time it was run [2]. Please have a look and post an updated version if necessary. == [1] https://commitfest.postgresql.org/46/4221/ [2] https://cirrus-ci.com/task/5618308

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-17 Thread Kartyshov Ivan
Add some fixes and rebase. -- Ivan Kartyshov Postgres Professional: www.postgrespro.comdiff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 4a42999b18..657a217e27 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -188,6 +188,7 @@ Com

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-11 Thread Kartyshov Ivan
Rebased and ready for review. I left only versions (due to irreparable problems) 1) Classic (wait_classic_v4.patch) https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru == advantages: multiple wait events, separate WAIT FOR statement disadvantages: new w

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-09 Thread vignesh C
On Fri, 8 Dec 2023 at 15:17, Kartyshov Ivan wrote: > > Should rise disscusion on separate utility statement or find > case where procedure version is failed. > > 1) Classic (wait_classic_v3.patch) > https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru > =

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-12-08 Thread Alexander Korotkov
On Fri, Dec 8, 2023 at 11:46 AM Kartyshov Ivan wrote: > > Should rise disscusion on separate utility statement or find > case where procedure version is failed. > > 1) Classic (wait_classic_v3.patch) > https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru > ==

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-12-08 Thread Alexander Korotkov
On Fri, Dec 8, 2023 at 11:20 AM Kartyshov Ivan wrote: > > On 2023-11-27 03:08, Alexander Korotkov wrote: > > I've retried my case with v6 and it doesn't fail anymore. But I > > wonder how safe it is to reset xmin within the user-visible function? > > We have no guarantee that the function is not

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-12-08 Thread Kartyshov Ivan
Should rise disscusion on separate utility statement or find case where procedure version is failed. 1) Classic (wait_classic_v3.patch) https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru == advantages: multiple wait events, separate WAIT FOR statement

  1   2   >