Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-11 Thread Heikki Linnakangas
On 11/04/2024 18:09, Alexander Korotkov wrote: On Thu, Apr 11, 2024 at 1:46 AM Heikki Linnakangas wrote: On 07/04/2024 00:52, Alexander Korotkov wrote: * At first, we check that pg_wal_replay_wait() is called in a non-atomic * context. That is, a procedure call isn't wrapped

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-11 Thread Alexander Korotkov
Hi, Heikki! Thank you for your interest in the subject. On Thu, Apr 11, 2024 at 1:46 AM Heikki Linnakangas wrote: > On 07/04/2024 00:52, Alexander Korotkov wrote: > > On Fri, Apr 5, 2024 at 9:15 PM Alvaro Herrera > > wrote: > >> I'm still concerned that WaitLSNCleanup is only called in

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-10 Thread Heikki Linnakangas
On 07/04/2024 00:52, Alexander Korotkov wrote: On Fri, Apr 5, 2024 at 9:15 PM Alvaro Herrera wrote: I'm still concerned that WaitLSNCleanup is only called in ProcKill. Does this mean that if a process throws an error while waiting, it'll not get cleaned up until it exits? Maybe this is not a

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-10 Thread Kartyshov Ivan
I did some experiments over synchronous replications and got that cascade replication can`t be synchronous. And  pg_wal_replay_wait() allows us to read your writes consistency on cascade replication. Beyond that, I added more tests on multi-standby replication and cascade replications. -- Ivan

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-06 Thread Alexander Korotkov
Hi, Alvaro! Thank you for your care on this matter. On Fri, Apr 5, 2024 at 9:15 PM Alvaro Herrera wrote: > BTW I noticed that > https://coverage.postgresql.org/src/backend/commands/waitlsn.c.gcov.html > says that lsn_cmp is not covered by the tests. This probably indicates > that the tests are

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-05 Thread Alvaro Herrera
Hello, BTW I noticed that https://coverage.postgresql.org/src/backend/commands/waitlsn.c.gcov.html says that lsn_cmp is not covered by the tests. This probably indicates that the tests are a little too light, but I'm not sure how much extra effort we want to spend. I'm still concerned that

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alexander Korotkov
On Wed, Apr 3, 2024 at 10:04 PM Pavel Borisov wrote: > On Wed, 3 Apr 2024 at 22:18, Alexander Korotkov wrote: >> >> On Wed, Apr 3, 2024 at 7:55 PM Alvaro Herrera >> wrote: >> > >> > On 2024-Apr-03, Alexander Korotkov wrote: >> > >> > > Regarding the shmem data structure for LSN waiters. I

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Pavel Borisov
Hi, Alexander! On Wed, 3 Apr 2024 at 22:18, Alexander Korotkov wrote: > On Wed, Apr 3, 2024 at 7:55 PM Alvaro Herrera > wrote: > > > > On 2024-Apr-03, Alexander Korotkov wrote: > > > > > Regarding the shmem data structure for LSN waiters. I didn't pick > > > LWLock or ConditionVariable,

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alexander Korotkov
On Wed, Apr 3, 2024 at 7:55 PM Alvaro Herrera wrote: > > On 2024-Apr-03, Alexander Korotkov wrote: > > > Regarding the shmem data structure for LSN waiters. I didn't pick > > LWLock or ConditionVariable, because I needed the ability to wake up > > only those waiters whose LSN is already

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alvaro Herrera
Hello Alexander, On 2024-Apr-03, Alexander Korotkov wrote: > Regarding the shmem data structure for LSN waiters. I didn't pick > LWLock or ConditionVariable, because I needed the ability to wake up > only those waiters whose LSN is already replayed. In my experience > waking up a process is

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alexander Korotkov
Hi, Alvaro! Thank you for your feedback. On Wed, Apr 3, 2024 at 9:58 AM Alvaro Herrera wrote: > Hello, I noticed that commit 06c418e163e9 uses waitLSN->mutex (a > spinlock) to protect the contents of waitLSN -- and it's used to walk an > arbitrary long list of processes waiting ... and also, an

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Andy Fan
> I also can confirm that a lot of users would be very happy to have > "read your writes consistency" and ready to do something to achieve > this at an application level. However, they typically don't know what > exactly they need. > > So, blogging about pg_wal_replay_wait() and spreading words

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Daniel Gustafsson
Buildfarm animal mamba (NetBSD 10.0 on macppc) started failing on this with what seems like a bogus compiler warning: waitlsn.c: In function 'WaitForLSN': waitlsn.c:275:24: error: 'endtime' may be used uninitialized in this function [-Werror=maybe-uninitialized] 275 |delay_ms = (endtime -

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-03 Thread Alvaro Herrera
Hello, I noticed that commit 06c418e163e9 uses waitLSN->mutex (a spinlock) to protect the contents of waitLSN -- and it's used to walk an arbitrary long list of processes waiting ... and also, an arbitrary number of processes could be calling this code. I think using a spinlock for this is

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Kartyshov Ivan
On 2024-04-02 13:15, Bharath Rupireddy wrote: On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan wrote: 8 years, we tried to add this feature, and now we suggest the best way for this feature is to commit the minimal version first. Just curious, do you or anyone else have an immediate use for

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Alexander Korotkov
On Tue, Apr 2, 2024 at 2:47 PM Andy Fan wrote: > Bharath Rupireddy writes: > > > On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan > > wrote: > >> > >> 8 years, we tried to add this feature, and now we suggest the best way > >> for this feature is to commit the minimal version first. > > > > Just

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Alexander Korotkov
On Tue, Apr 2, 2024 at 1:11 PM Kartyshov Ivan wrote: > On 2024-04-02 11:14, Andy Fan wrote: > > If so, when users call pg_wal_replay_wait, they can get informed when > > the wal is replaied to the target_lsn, but they can't know how long > > time > > it waits unless they check it in application

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Andy Fan
Bharath Rupireddy writes: > On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan > wrote: >> >> 8 years, we tried to add this feature, and now we suggest the best way >> for this feature is to commit the minimal version first. > > Just curious, do you or anyone else have an immediate use for this >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Bharath Rupireddy
On Tue, Apr 2, 2024 at 3:41 PM Kartyshov Ivan wrote: > > 8 years, we tried to add this feature, and now we suggest the best way > for this feature is to commit the minimal version first. Just curious, do you or anyone else have an immediate use for this function? If yes, how are they achieving

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Kartyshov Ivan
On 2024-04-02 11:14, Andy Fan wrote: If so, when users call pg_wal_replay_wait, they can get informed when the wal is replaied to the target_lsn, but they can't know how long time it waits unless they check it in application side, I think such information will be useful for monitor purpose

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Andy Fan
Hi Alexander! >> + >> +pg_wal_replay_wait ( >> + target_lsn pg_lsn, >> + timeout bigint >> DEFAULT 0) >> +void >> + >> >> Should we return the millseconds of waiting time? I think this >> information may be useful for customer if they want to

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-02 Thread Alexander Korotkov
Hi, Andy! On Tue, Apr 2, 2024 at 6:29 AM Andy Fan wrote: > > Did you forget to attach the new patch? > > > > Yes, here it is. > > > > [4. text/x-diff; > > v17-0001-Implement-pg_wal_replay_wait-stored-procedure.patch]... > > + > +pg_wal_replay_wait ( > + target_lsn

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-01 Thread Andy Fan
Alexander Korotkov writes: Hello, > > Did you forget to attach the new patch? > > Yes, here it is. > > -- > Regards, > Alexander Korotkov > > [4. text/x-diff; > v17-0001-Implement-pg_wal_replay_wait-stored-procedure.patch]... + +pg_wal_replay_wait ( +

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-01 Thread Alexander Korotkov
On Mon, Apr 1, 2024 at 5:25 AM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > On Mon, Apr 1, 2024 at 5:54 AM Alexander Korotkov > wrote: > > > > > 9. To me the following query blocks even though I didn't mention > timeout. > > > CALL pg_wal_replay_wait('0/fff'); > > > >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-31 Thread Bharath Rupireddy
On Mon, Apr 1, 2024 at 5:54 AM Alexander Korotkov wrote: > > > 9. To me the following query blocks even though I didn't mention timeout. > > CALL pg_wal_replay_wait('0/fff'); > > If your primary server is freshly initialized, you need to do quite > data modifications to reach this LSN.

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-31 Thread Alexander Korotkov
Hi Bharath, Thank you for your feedback. On Sun, Mar 31, 2024 at 8:44 AM Bharath Rupireddy wrote: > On Sun, Mar 31, 2024 at 7:41 AM Alexander Korotkov > wrote: > Thanks for the patch. I have a few comments on the v16 patch. > > 1. Is there any specific reason for pg_wal_replay_wait() being a

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Bharath Rupireddy
On Sun, Mar 31, 2024 at 7:41 AM Alexander Korotkov wrote: > > Hi! Thanks for the patch. I have a few comments on the v16 patch. 1. Is there any specific reason for pg_wal_replay_wait() being a procedure rather than a function? I haven't read the full thread, but I vaguely noticed the discussion

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Alexander Korotkov
Hi! On Sat, Mar 30, 2024 at 6:14 PM Kartyshov Ivan wrote: > > Thank you Alexander for working on patch, may be we should change some > names: > 1) test 043_wait_lsn.pl -> to 043_waitlsn.pl like waitlsn.c and > waitlsn.h I renamed that to 043_wal_replay_wait.pl to match the name of SQL

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Kartyshov Ivan
Thank you Alexander for working on patch, may be we should change some names: 1) test 043_wait_lsn.pl -> to 043_waitlsn.pl like waitlsn.c and waitlsn.h In waitlsn.c and waitlsn.h variables: 2) targret_lsn -> trgLSN like curLSN 3) lsn -> trgLSN like curLSN -- Ivan Kartyshov Postgres

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-30 Thread Alexander Korotkov
On Fri, Mar 29, 2024 at 6:50 PM Euler Taveira wrote: > On Fri, Mar 29, 2024, at 9:44 AM, Alexander Korotkov wrote: > > This generally makes sense, but I'm not sure about this. The > milliseconds timeout was used initially but received critics in [1]. > > > Alexander, I see why you changed the

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Euler Taveira
On Fri, Mar 29, 2024, at 9:44 AM, Alexander Korotkov wrote: > This generally makes sense, but I'm not sure about this. The > milliseconds timeout was used initially but received critics in [1]. Alexander, I see why you changed the patch. Peter suggested to use an interval but you proposed

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Pavel Borisov
Hi, hackers! On Fri, 29 Mar 2024 at 16:45, Alexander Korotkov wrote: > Hi, Euler! > > On Fri, Mar 29, 2024 at 1:38 AM Euler Taveira wrote: > > On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > > > > Fixed along with other issues spotted by Alexander Lakhin. > > > > > > [I didn't

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-29 Thread Alexander Korotkov
Hi, Euler! On Fri, Mar 29, 2024 at 1:38 AM Euler Taveira wrote: > On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > > Fixed along with other issues spotted by Alexander Lakhin. > > > [I didn't read the whole thread. I'm sorry if I missed something ...] > > You renamed the function in

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Euler Taveira
On Thu, Mar 28, 2024, at 9:39 AM, Alexander Korotkov wrote: > Fixed along with other issues spotted by Alexander Lakhin. [I didn't read the whole thread. I'm sorry if I missed something ...] You renamed the function in a previous version but let me suggest another one: pg_wal_replay_wait. It

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Alexander Korotkov
On Fri, Mar 22, 2024 at 3:45 PM Kartyshov Ivan wrote: > On 2024-03-20 12:11, Alexander Korotkov wrote: > > On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan > > wrote: > >> > 4.2 With an unreasonably high future LSN, BEGIN command waits > >> > unboundedly, shouldn't we check if the specified LSN

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Alexander Korotkov
On Thu, Mar 28, 2024 at 9:37 AM Thomas Munro wrote: > > > v12 > > Hi all, > > I didn't review the patch but one thing jumped out: I don't think it's > OK to hold a spinlock while (1) looping over an array of backends and > (2) making system calls (SetLatch()). Good catch, thank you. Fixed along

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Thomas Munro
> v12 Hi all, I didn't review the patch but one thing jumped out: I don't think it's OK to hold a spinlock while (1) looping over an array of backends and (2) making system calls (SetLatch()).

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-28 Thread Alexander Korotkov
On Thu, Mar 28, 2024 at 2:24 AM Alexander Korotkov wrote: > On Tue, Mar 26, 2024 at 4:06 PM Kartyshov Ivan > wrote: > > Thank you for your interest to the patch. > > I understand you questions, but I fully support Alexander Korotkov idea > > to commit the minimal required functionality. And then

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-27 Thread Alexander Korotkov
On Tue, Mar 26, 2024 at 4:06 PM Kartyshov Ivan wrote: > Thank you for your interest to the patch. > I understand you questions, but I fully support Alexander Korotkov idea > to commit the minimal required functionality. And then keep working on > other improvements. I did further improvements in

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Alexander Korotkov
On Sun, Mar 24, 2024 at 4:39 AM Bharath Rupireddy wrote: > I share the same concern as yours and had proposed something upthread > [1]. The idea is something like how each query takes a snapshot at the > beginning of txn/query (depending on isolation level), the same way > the standby can wait

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Alexander Korotkov
On Fri, Mar 22, 2024 at 12:58 AM Peter Eisentraut wrote: > On 17.03.24 15:09, Alexander Korotkov wrote: > > My current attempt was to commit minimal implementation as less > > invasive as possible. A new clause for BEGIN doesn't require > > additional keywords and doesn't introduce additional

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Alexander Korotkov
On Fri, Mar 22, 2024 at 12:50 AM Peter Eisentraut wrote: > On 19.03.24 18:38, Kartyshov Ivan wrote: > > CALL pg_wait_lsn('0/3002AE8', 1); > > BEGIN; > > SELECT * FROM tbl; // read fresh insertions > > COMMIT; > > I'm not endorsing this

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-26 Thread Kartyshov Ivan
Thank you for your interest to the patch. I understand you questions, but I fully support Alexander Korotkov idea to commit the minimal required functionality. And then keep working on other improvements. On 2024-03-24 05:39, Bharath Rupireddy wrote: On Fri, Mar 22, 2024 at 4:28 AM Peter

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-23 Thread Bharath Rupireddy
On Fri, Mar 22, 2024 at 4:28 AM Peter Eisentraut wrote: > > I had written in [0] about my questions related to using this with > connection poolers. I don't think this was addressed at all. I haven't > seen any discussion about how to make this kind of facility usable in a > full system. You

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-22 Thread Kartyshov Ivan
Thank you for your feedback. On 2024-03-20 12:11, Alexander Korotkov wrote: On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > 4.2 With an unreasonably high future LSN, BEGIN command waits > unboundedly, shouldn't we check if the specified LSN is more than > pg_last_wal_receive_lsn()

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-22 Thread Kartyshov Ivan
Thank you for your feedback. On 2024-03-20 12:11, Alexander Korotkov wrote: On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > 4.2 With an unreasonably high future LSN, BEGIN command waits > unboundedly, shouldn't we check if the specified LSN is more than > pg_last_wal_receive_lsn()

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-22 Thread Kartyshov Ivan
On 2024-03-20 12:11, Alexander Korotkov wrote: On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > 4.2 With an unreasonably high future LSN, BEGIN command waits > unboundedly, shouldn't we check if the specified LSN is more than > pg_last_wal_receive_lsn() error out? I think limiting

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-21 Thread Peter Eisentraut
On 17.03.24 15:09, Alexander Korotkov wrote: My current attempt was to commit minimal implementation as less invasive as possible. A new clause for BEGIN doesn't require additional keywords and doesn't introduce additional statements. But yes, this is still a new qual. And, yes, Amit you're

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-21 Thread Peter Eisentraut
On 19.03.24 18:38, Kartyshov Ivan wrote: CALL pg_wait_lsn('0/3002AE8', 1); BEGIN; SELECT * FROM tbl; // read fresh insertions COMMIT; I'm not endorsing this or any other approach, but I think the timeout parameter should be of type

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-20 Thread Alexander Korotkov
On Wed, Mar 20, 2024 at 12:34 AM Kartyshov Ivan wrote: > > 4.2 With an unreasonably high future LSN, BEGIN command waits > > unboundedly, shouldn't we check if the specified LSN is more than > > pg_last_wal_receive_lsn() error out? I think limiting wait lsn by current received lsn would destroy

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Kartyshov Ivan
Bharath Rupireddy, thank you for you review. But here is some points. On 2024-03-16 10:02, Bharath Rupireddy wrote: 4.1 With invalid LSN succeeds, shouldn't it error out? Or at least, add a fast path/quick exit to WaitForLSN()? BEGIN AFTER '0/0'; In postgresql '0/0' is Valid pg_lsn, but it is

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Kartyshov Ivan
Intro == The main purpose of the feature is to achieve read-your-writes-consistency, while using async replica for reads and primary for writes. In that case lsn of last modification is stored inside application. We cannot store this lsn inside database, since reads are distributed across

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Alexander Korotkov
On Tue, Mar 19, 2024 at 1:51 PM Amit Kapila wrote: > On Mon, Mar 18, 2024 at 3:24 PM Alexander Korotkov > wrote: > > > > On Mon, Mar 18, 2024 at 5:17 AM Amit Kapila wrote: > > > > 1. First, check that it was called with non-atomic context (that is, > > > > it's not called within a

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-19 Thread Amit Kapila
On Mon, Mar 18, 2024 at 3:24 PM Alexander Korotkov wrote: > > On Mon, Mar 18, 2024 at 5:17 AM Amit Kapila wrote: > > > 1. First, check that it was called with non-atomic context (that is, > > > it's not called within a transaction). Trigger error if called with > > > atomic context. > > > 2.

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-18 Thread Alexander Korotkov
On Mon, Mar 18, 2024 at 5:17 AM Amit Kapila wrote: > On Sun, Mar 17, 2024 at 7:40 PM Alexander Korotkov wrote: > > On Sat, Mar 16, 2024 at 5:05 PM Bharath Rupireddy > > wrote: > > > On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > > > In general, it seems this patch has been stuck for a

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-17 Thread Amit Kapila
On Sun, Mar 17, 2024 at 7:40 PM Alexander Korotkov wrote: > > On Sat, Mar 16, 2024 at 5:05 PM Bharath Rupireddy > wrote: > > On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > > In general, it seems this patch has been stuck for a long time on the > > > decision to choose an appropriate UI

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-17 Thread Alexander Korotkov
Hi Amit, Hi Bharath, On Sat, Mar 16, 2024 at 5:05 PM Bharath Rupireddy wrote: > On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > In general, it seems this patch has been stuck for a long time on the > > decision to choose an appropriate UI (syntax), and we thought of > > moving it further

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-16 Thread Bharath Rupireddy
On Sat, Mar 16, 2024 at 4:26 PM Amit Kapila wrote: > > On Fri, Mar 15, 2024 at 7:50 PM Alexander Korotkov > wrote: > > > > I went through this patch another time, and made some minor > > adjustments. Now it looks good, I'm going to push it if no > > objections. > > > > I have a question

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-16 Thread Amit Kapila
On Fri, Mar 15, 2024 at 7:50 PM Alexander Korotkov wrote: > > I went through this patch another time, and made some minor > adjustments. Now it looks good, I'm going to push it if no > objections. > I have a question related to usability, if the regular reads (say a Select statement or reads

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-16 Thread Bharath Rupireddy
On Sat, Mar 16, 2024 at 2:04 AM Alexander Korotkov wrote: > > > Rebase and update patch. Thanks for working on this. I took a quick look at v11 patch. Here are some comments: 1. +#include "utils/timestamp.h" +#include "executor/spi.h" +#include "utils/fmgrprotos.h" Please place executor/spi.h

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Alexander Korotkov
On Fri, Mar 15, 2024 at 10:32 PM Kartyshov Ivan wrote: > > On 2024-03-15 22:59, Kartyshov Ivan wrote: > > On 2024-03-11 13:44, Alexander Korotkov wrote: > >> I picked the second option and left only the AFTER clause for the > >> BEGIN statement. I think this should be enough for the beginning. >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Kartyshov Ivan
On 2024-03-15 22:59, Kartyshov Ivan wrote: On 2024-03-11 13:44, Alexander Korotkov wrote: I picked the second option and left only the AFTER clause for the BEGIN statement. I think this should be enough for the beginning. Thank you for your rework on your patch, here I made some fixes: 0)

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Kartyshov Ivan
On 2024-03-11 13:44, Alexander Korotkov wrote: I picked the second option and left only the AFTER clause for the BEGIN statement. I think this should be enough for the beginning. Thank you for your rework on your patch, here I made some fixes: 0) autocomplete 1) less jumps 2) more description

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Alexander Korotkov
On Fri, Mar 15, 2024 at 4:20 PM Alexander Korotkov wrote: > On Mon, Mar 11, 2024 at 12:44 PM Alexander Korotkov > wrote: > > I've decided to put my hands on this patch. > > > > On Thu, Mar 7, 2024 at 2:25 PM Amit Kapila > wrote: > > > +1 for the second one not only because it avoids new words

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-15 Thread Alexander Korotkov
On Mon, Mar 11, 2024 at 12:44 PM Alexander Korotkov wrote: > I've decided to put my hands on this patch. > > On Thu, Mar 7, 2024 at 2:25 PM Amit Kapila wrote: > > +1 for the second one not only because it avoids new words in grammar > > but also sounds to convey the meaning. I think you can

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-11 Thread Alexander Korotkov
Hi! I've decided to put my hands on this patch. On Thu, Mar 7, 2024 at 2:25 PM Amit Kapila wrote: > +1 for the second one not only because it avoids new words in grammar > but also sounds to convey the meaning. I think you can explain in docs > how this feature can be used basically how will

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-07 Thread Amit Kapila
On Thu, Mar 7, 2024 at 5:14 PM Kartyshov Ivan wrote: > > Intro > == > The main purpose of the feature is to achieve > read-your-writes-consistency, while using async replica for reads and > primary for writes. In that case lsn of last modification is stored > inside application. We cannot

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-03-07 Thread Kartyshov Ivan
Intro == The main purpose of the feature is to achieve read-your-writes-consistency, while using async replica for reads and primary for writes. In that case lsn of last modification is stored inside application. We cannot store this lsn inside database, since reads are distributed across

Fwd: Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-02-01 Thread Kartyshov Ivan
Updated, rebased, fixed Ci and added documentation. We left two different solutions. Help me please to choose the best. 1) Classic (wait_classic_v6.patch) https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru == advantages: multiple wait events, separate

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-21 Thread Dilip Kumar
On Wed, Jan 17, 2024 at 1:46 PM Kartyshov Ivan wrote: > > Add some fixes and rebase. > While quickly looking into the patch, I understood the idea of what we are trying to achieve here and I feel that a useful feature. But while looking at both the patches I could not quickly differentiate

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-21 Thread Peter Smith
2024-01 Commitfest. Hi, This patch has a CF status of "Needs Review" [1], but it seems there was a CFbot test failure last time it was run [2]. Please have a look and post an updated version if necessary. == [1] https://commitfest.postgresql.org/46/4221/ [2]

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-17 Thread Kartyshov Ivan
Add some fixes and rebase. -- Ivan Kartyshov Postgres Professional: www.postgrespro.comdiff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 4a42999b18..657a217e27 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -188,6 +188,7 @@

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-11 Thread Kartyshov Ivan
Rebased and ready for review. I left only versions (due to irreparable problems) 1) Classic (wait_classic_v4.patch) https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru == advantages: multiple wait events, separate WAIT FOR statement disadvantages: new

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-09 Thread vignesh C
On Fri, 8 Dec 2023 at 15:17, Kartyshov Ivan wrote: > > Should rise disscusion on separate utility statement or find > case where procedure version is failed. > > 1) Classic (wait_classic_v3.patch) > https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-12-08 Thread Alexander Korotkov
On Fri, Dec 8, 2023 at 11:46 AM Kartyshov Ivan wrote: > > Should rise disscusion on separate utility statement or find > case where procedure version is failed. > > 1) Classic (wait_classic_v3.patch) > https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-12-08 Thread Alexander Korotkov
On Fri, Dec 8, 2023 at 11:20 AM Kartyshov Ivan wrote: > > On 2023-11-27 03:08, Alexander Korotkov wrote: > > I've retried my case with v6 and it doesn't fail anymore. But I > > wonder how safe it is to reset xmin within the user-visible function? > > We have no guarantee that the function is not

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-12-08 Thread Kartyshov Ivan
Should rise disscusion on separate utility statement or find case where procedure version is failed. 1) Classic (wait_classic_v3.patch) https://www.postgresql.org/message-id/3cc883048264c2e9af022033925ff8db%40postgrespro.ru == advantages: multiple wait events, separate WAIT FOR statement

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-12-08 Thread Kartyshov Ivan
On 2023-11-27 03:08, Alexander Korotkov wrote: I've retried my case with v6 and it doesn't fail anymore. But I wonder how safe it is to reset xmin within the user-visible function? We have no guarantee that the function is not called inside the complex query. Then how will the rest of the

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-11-26 Thread Alexander Korotkov
On Mon, Nov 20, 2023 at 1:10 PM Картышов Иван wrote: > Alexander, thank you for your review and pointing this issues. According to > them I made some fixes and rebase all patch. > > But I can`t repeat your ERROR. Not with hot_standby_feedback = on nor > hot_standby_feedback = off. > > master:

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-11-26 Thread Alexander Korotkov
On Thu, Nov 23, 2023 at 5:52 AM Bowen Shi wrote: > I used the latest code and found some conflicts while applying. Which PG > version did you rebase? I've successfully applied the patch on bc3c8db8ae. But I've used "patch -p1 < wait_proc_v6.patch", git am doesn't work. -- Regards,

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-11-22 Thread Bowen Shi
Hi, I used the latest code and found some conflicts while applying. Which PG version did you rebase? Regards Bowen Shi

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-11-20 Thread Картышов Иван
Alexander, thank you for your review and pointing this issues. According to them I made some fixes and rebase all patch. But I can`t repeat your ERROR. Not with hot_standby_feedback = on nor  hot_standby_feedback = off.master: create table test as (select i from generate_series(1,1) i);

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-10-14 Thread Alexander Korotkov
Hi! On Wed, Oct 4, 2023 at 1:22 PM Alexander Korotkov wrote: > I see you're concentrating on the procedural version of this feature. But > when you're calling a procedure within a normal SQL statement, the executor > gets a snapshot and holds it until the procedure finishes. In the case the

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-10-04 Thread Alexander Korotkov
Hi, Ivan! On Fri, Jun 30, 2023 at 11:32 AM Картышов Иван wrote: > All rebased and tested > Thank you for continuing to work on this patch. I see you're concentrating on the procedural version of this feature. But when you're calling a procedure within a normal SQL statement, the executor

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-06-30 Thread Картышов Иван
All rebased and tested -- Ivan Kartyshov Postgres Professional: http://www.postgrespro.com The Russian Postgres comp...@postgrespro.ru>   diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index becc2bda62..c7460bd9b8 100644 ---

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-03-06 Thread Kartyshov Ivan
Fix build.meson troubles -- Ivan Kartyshov Postgres Professional: http://www.postgrespro.com The Russian Postgres Companydiff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index dbe9394762..422bb1ed82 100644 ---

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-03-06 Thread Kartyshov Ivan
Update patch to fix conflict with master -- Ivan Kartyshov Postgres Professional: http://www.postgrespro.com The Russian Postgres Companydiff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index dbe9394762..422bb1ed82 100644 ---

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-03-04 Thread Kartyshov Ivan
Here I made new patch of feature, discussed above. WAIT FOR procedure - waits for certain lsn on pause == Synopsis == SELECT pg_wait_lsn(‘LSN’, timeout) returns boolean Where timeout = 0, will wait infinite without timeout And if timeout = 1, then just check if lsn

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-03-02 Thread Peter Eisentraut
On 28.02.23 11:10, Kartyshov Ivan wrote: 3) Procedure style: Tom Lane and Kyotaro (wait_proc_v1.patch) https://www.postgresql.org/message-id/27171.1586439221%40sss.pgh.pa.us https://www.postgresql.org/message-id/20210121.173009.235021120161403875.horikyota.ntt%40gmail.com == advantages:

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-03-01 Thread Michael Paquier
On Wed, Mar 01, 2023 at 03:31:06PM -0500, Greg Stark wrote: > Fwiw I prefer the functions approach. I do like me some nice syntax > but I don't see any particular advantage of the special syntax in this > case. They don't seem to provide any additional expressiveness. So do I, eventhough I saw a

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-03-01 Thread Greg Stark
On Tue, 28 Feb 2023 at 05:13, Kartyshov Ivan wrote: > > Below I provide the implementation of patches for the first three types. > I propose to discuss this feature again/ Oof, that doesn't really work with the cfbot. It tries to apply all three patches and of course the second and third fail to

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-02-28 Thread Kartyshov Ivan
Intro== The main purpose of the feature is to achieve read-your-writes-consistency, while using async replica for reads and primary for writes. In that case lsn of last modification is stored inside application. We cannot store this lsn inside database, since reads are distributed

Re: [HACKERS] make async slave to wait for lsn to be replayed

2021-03-21 Thread Kyotaro Horiguchi
At Thu, 18 Mar 2021 18:57:15 +0500, Ibrar Ahmed wrote in > On Thu, Jan 21, 2021 at 1:30 PM Kyotaro Horiguchi > wrote: > > > Hello. > > > > At Wed, 18 Nov 2020 15:05:00 +0300, a.pervush...@postgrespro.ru wrote in > > > I've changed the BEGIN WAIT FOR LSN statement to core functions > > >

Re: [HACKERS] make async slave to wait for lsn to be replayed

2021-03-18 Thread Ibrar Ahmed
On Thu, Jan 21, 2021 at 1:30 PM Kyotaro Horiguchi wrote: > Hello. > > At Wed, 18 Nov 2020 15:05:00 +0300, a.pervush...@postgrespro.ru wrote in > > I've changed the BEGIN WAIT FOR LSN statement to core functions > > pg_waitlsn, pg_waitlsn_infinite and pg_waitlsn_no_wait. > > Currently the

Re: [HACKERS] make async slave to wait for lsn to be replayed

2021-01-21 Thread Kyotaro Horiguchi
Hello. At Wed, 18 Nov 2020 15:05:00 +0300, a.pervush...@postgrespro.ru wrote in > I've changed the BEGIN WAIT FOR LSN statement to core functions > pg_waitlsn, pg_waitlsn_infinite and pg_waitlsn_no_wait. > Currently the functions work inside repeatable read transactions, but > waitlsn creates a

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-11-18 Thread a . pervushina
Hello, I've changed the BEGIN WAIT FOR LSN statement to core functions pg_waitlsn, pg_waitlsn_infinite and pg_waitlsn_no_wait. Currently the functions work inside repeatable read transactions, but waitlsn creates a snapshot if called first in a transaction block, which can possibly lead the

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-11-16 Thread Alexander Korotkov
Hi! On Mon, Nov 16, 2020 at 1:09 PM wrote: > I've changed the BEGIN WAIT FOR LSN statement to core functions > pg_waitlsn, pg_waitlsn_infinite and pg_waitlsn_no_wait. > Currently the functions work inside repeatable read transactions, but > waitlsn creates a snapshot if called first in a

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-11-16 Thread a . pervushina
Hello, I've changed the BEGIN WAIT FOR LSN statement to core functions pg_waitlsn, pg_waitlsn_infinite and pg_waitlsn_no_wait. Currently the functions work inside repeatable read transactions, but waitlsn creates a snapshot if called first in a transaction block, which can possibly lead the

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-10-02 Thread a . pervushina
Anna Akenteva писал 2020-04-08 22:36: On 2020-04-08 04:09, Kyotaro Horiguchi wrote: I like your suggested keywords! I think that "AFTER" + "WITHIN" sound the most natural. We could completely give up the LSN keyword for now. The final command could look something like: BEGIN AFTER ‘0/303EC60’

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-09-23 Thread Michael Paquier
On Tue, Aug 18, 2020 at 01:12:51PM +0300, Anna Akenteva wrote: > I updated the most recent patch and removed the use of "master" from it, > replacing it with "primary". This is failing to apply lately, causing the CF bot to complain: http://cfbot.cputube.org/patch_29_772.log -- Michael

  1   2   >