Re: Replication slot stats misgivings

2021-05-25 Thread Amit Kapila
On Mon, May 24, 2021 at 10:09 AM vignesh C wrote: > > On Mon, May 24, 2021 at 9:38 AM Amit Kapila wrote: > > > > On Thu, May 13, 2021 at 11:30 AM vignesh C wrote: > > > > > > > Do we want to update the information about pg_stat_replication_slots > > at the following place in docs > >

Re: Replication slot stats misgivings

2021-05-23 Thread vignesh C
On Mon, May 24, 2021 at 9:38 AM Amit Kapila wrote: > > On Thu, May 13, 2021 at 11:30 AM vignesh C wrote: > > > > Do we want to update the information about pg_stat_replication_slots > at the following place in docs > https://www.postgresql.org/docs/devel/logicaldecoding-catalogs.html? > > If so,

Re: Replication slot stats misgivings

2021-05-23 Thread Amit Kapila
On Thu, May 13, 2021 at 11:30 AM vignesh C wrote: > Do we want to update the information about pg_stat_replication_slots at the following place in docs https://www.postgresql.org/docs/devel/logicaldecoding-catalogs.html? If so, feel free to submit the patch for it? -- With Regards, Amit

Re: Replication slot stats misgivings

2021-05-13 Thread vignesh C
On Thu, May 13, 2021 at 11:21 AM Amit Kapila wrote: > > On Wed, May 12, 2021 at 4:02 PM Masahiko Sawada wrote: > > > > On Wed, May 12, 2021 at 1:19 PM vignesh C wrote: > > > > > > > I think the part of the test that tests the stats after resetting it > > > > might give different results. This

Re: Replication slot stats misgivings

2021-05-12 Thread Amit Kapila
On Wed, May 12, 2021 at 4:02 PM Masahiko Sawada wrote: > > On Wed, May 12, 2021 at 1:19 PM vignesh C wrote: > > > > > I think the part of the test that tests the stats after resetting it > > > might give different results. This can happen because in the previous > > > test we spill multiple

Re: Replication slot stats misgivings

2021-05-12 Thread Masahiko Sawada
On Wed, May 12, 2021 at 1:19 PM vignesh C wrote: > > On Wed, May 12, 2021 at 9:08 AM Amit Kapila wrote: > > > > On Wed, May 12, 2021 at 7:53 AM Amit Kapila wrote: > > > > > > On Wed, May 12, 2021 at 4:00 AM Masahiko Sawada > > > wrote: > > > > > > > > Ugh, since by commit 592f00f8de we send

Re: Replication slot stats misgivings

2021-05-11 Thread Tom Lane
vignesh C writes: > On Wed, May 12, 2021 at 9:59 AM Tom Lane wrote: >> Is there any value in converting the test case into a TAP test that >> could be more flexible about the expected output? I'm mainly wondering >> whether there are any code paths that this test forces the server through, >>

Re: Replication slot stats misgivings

2021-05-11 Thread vignesh C
On Wed, May 12, 2021 at 9:59 AM Tom Lane wrote: > > vignesh C writes: > > I agree with your analysis to remove that test. Attached patch has the > > changes for the same. > > Is there any value in converting the test case into a TAP test that > could be more flexible about the expected output?

Re: Replication slot stats misgivings

2021-05-11 Thread Tom Lane
vignesh C writes: > I agree with your analysis to remove that test. Attached patch has the > changes for the same. Is there any value in converting the test case into a TAP test that could be more flexible about the expected output? I'm mainly wondering whether there are any code paths that

Re: Replication slot stats misgivings

2021-05-11 Thread vignesh C
On Wed, May 12, 2021 at 9:08 AM Amit Kapila wrote: > > On Wed, May 12, 2021 at 7:53 AM Amit Kapila wrote: > > > > On Wed, May 12, 2021 at 4:00 AM Masahiko Sawada > > wrote: > > > > > > Ugh, since by commit 592f00f8de we send slot stats every after > > > spil/stream it’s possible that we report

Re: Replication slot stats misgivings

2021-05-11 Thread Amit Kapila
On Wed, May 12, 2021 at 7:53 AM Amit Kapila wrote: > > On Wed, May 12, 2021 at 4:00 AM Masahiko Sawada wrote: > > > > Ugh, since by commit 592f00f8de we send slot stats every after > > spil/stream it’s possible that we report slot stats that have non-zero > > counters for spill_bytes/txns and

Re: Replication slot stats misgivings

2021-05-11 Thread Amit Kapila
On Wed, May 12, 2021 at 4:00 AM Masahiko Sawada wrote: > > On Wed, May 12, 2021 at 6:32 AM Tom Lane wrote: > > > > Amit Kapila writes: > > > I have closed this open item. > > > > That seems a little premature, considering that the > > contrib/test_decoding/sql/stats.sql test case is still

Re: Replication slot stats misgivings

2021-05-11 Thread Masahiko Sawada
On Wed, May 12, 2021 at 6:32 AM Tom Lane wrote: > > Amit Kapila writes: > > I have closed this open item. > > That seems a little premature, considering that the > contrib/test_decoding/sql/stats.sql test case is still failing regularly. Thank you for reporting. Ugh, since by commit 592f00f8de

Re: Replication slot stats misgivings

2021-05-11 Thread Tom Lane
Amit Kapila writes: > I have closed this open item. That seems a little premature, considering that the contrib/test_decoding/sql/stats.sql test case is still failing regularly. https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=locust=2021-05-11%2019%3A14%3A53

Re: Replication slot stats misgivings

2021-05-10 Thread Amit Kapila
On Fri, May 7, 2021 at 8:03 AM Amit Kapila wrote: > > Thanks for the summarization. I don't find anything that is left > unaddressed. I think we can wait for a day or two to see if Andres or > anyone else sees anything that is left unaddressed and then we can > close the open item. > I have

Re: Replication slot stats misgivings

2021-05-06 Thread Amit Kapila
On Fri, May 7, 2021 at 6:10 AM Masahiko Sawada wrote: > > On Sat, Mar 20, 2021 at 3:52 AM Andres Freund wrote: > > > > Hi, > > > > I started to write this as a reply to > > https://postgr.es/m/20210318015105.dcfa4ceybdjubf2i%40alap3.anarazel.de > > but I think it doesn't really fit under that

Re: Replication slot stats misgivings

2021-05-06 Thread Masahiko Sawada
On Sat, Mar 20, 2021 at 3:52 AM Andres Freund wrote: > > Hi, > > I started to write this as a reply to > https://postgr.es/m/20210318015105.dcfa4ceybdjubf2i%40alap3.anarazel.de > but I think it doesn't really fit under that header anymore. > > On 2021-03-17 18:51:05 -0700, Andres Freund wrote: >

Re: Replication slot stats misgivings

2021-05-06 Thread Amit Kapila
On Thu, May 6, 2021 at 2:05 PM Masahiko Sawada wrote: > > On Thu, May 6, 2021 at 5:28 PM Amit Kapila wrote: > > > > On Thu, May 6, 2021 at 1:30 PM Masahiko Sawada > > wrote: > > > > > > All issues pointed out in this thread are resolved and we can remove > > > this item from the open items? >

Re: Replication slot stats misgivings

2021-05-06 Thread Masahiko Sawada
On Thu, May 6, 2021 at 5:28 PM Amit Kapila wrote: > > On Thu, May 6, 2021 at 1:30 PM Masahiko Sawada wrote: > > > > All issues pointed out in this thread are resolved and we can remove > > this item from the open items? > > > > I think so. Do you think we should reply to Andres's original email

Re: Replication slot stats misgivings

2021-05-06 Thread vignesh C
On Thu, May 6, 2021 at 1:30 PM Masahiko Sawada wrote: > > On Thu, May 6, 2021 at 4:03 PM Amit Kapila wrote: > > > > On Thu, May 6, 2021 at 10:55 AM Masahiko Sawada > > wrote: > > > > > > On Thu, May 6, 2021 at 1:09 PM Amit Kapila > > > wrote: > > > > > > > > > > > In the attached, I have

Re: Replication slot stats misgivings

2021-05-06 Thread Amit Kapila
On Thu, May 6, 2021 at 1:30 PM Masahiko Sawada wrote: > > All issues pointed out in this thread are resolved and we can remove > this item from the open items? > I think so. Do you think we should reply to Andres's original email stating the commits that fixed the individual review comments to

Re: Replication slot stats misgivings

2021-05-06 Thread Masahiko Sawada
On Thu, May 6, 2021 at 4:03 PM Amit Kapila wrote: > > On Thu, May 6, 2021 at 10:55 AM Masahiko Sawada wrote: > > > > On Thu, May 6, 2021 at 1:09 PM Amit Kapila wrote: > > > > > > > > In the attached, I have combined > > > Vignesh's patch and your doc fix patch. Additionally, I have changed > >

Re: Replication slot stats misgivings

2021-05-06 Thread Amit Kapila
On Thu, May 6, 2021 at 10:55 AM Masahiko Sawada wrote: > > On Thu, May 6, 2021 at 1:09 PM Amit Kapila wrote: > > > > > In the attached, I have combined > > Vignesh's patch and your doc fix patch. Additionally, I have changed > > some comments and some other cosmetic stuff. Let me know what you

Re: Replication slot stats misgivings

2021-05-05 Thread vignesh C
On Thu, May 6, 2021 at 9:39 AM Amit Kapila wrote: > > On Thu, May 6, 2021 at 6:15 AM Masahiko Sawada wrote: > > > > After more thought, I'm concerned that my patch's approach might be > > invasive for PG14. Given that Vignesh’s patch would cover most cases, > > > > I am not sure if your patch is

Re: Replication slot stats misgivings

2021-05-05 Thread Masahiko Sawada
On Thu, May 6, 2021 at 1:09 PM Amit Kapila wrote: > > On Thu, May 6, 2021 at 6:15 AM Masahiko Sawada wrote: > > > > After more thought, I'm concerned that my patch's approach might be > > invasive for PG14. Given that Vignesh’s patch would cover most cases, > > > > I am not sure if your patch is

Re: Replication slot stats misgivings

2021-05-05 Thread Amit Kapila
On Thu, May 6, 2021 at 6:15 AM Masahiko Sawada wrote: > > After more thought, I'm concerned that my patch's approach might be > invasive for PG14. Given that Vignesh’s patch would cover most cases, > I am not sure if your patch is too invasive but OTOH I am also convinced that Vignesh's patch

Re: Replication slot stats misgivings

2021-05-05 Thread Masahiko Sawada
On Mon, May 3, 2021 at 9:18 PM Masahiko Sawada wrote: > > On Mon, May 3, 2021 at 2:27 PM Amit Kapila wrote: > > > Apart from this, I think you > > have suggested somewhere in this thread to slightly update the > > description of stream_bytes. I would like to update the description of > >

Re: Replication slot stats misgivings

2021-05-05 Thread Masahiko Sawada
On Tue, May 4, 2021 at 2:34 PM vignesh C wrote: > > On Tue, May 4, 2021 at 9:48 AM Masahiko Sawada wrote: > > > > On Mon, May 3, 2021 at 10:21 PM Amit Kapila wrote: > > > > > > On Mon, May 3, 2021 at 5:48 PM Masahiko Sawada > > > wrote: > > > > > > > > On Mon, May 3, 2021 at 2:27 PM Amit

Re: Replication slot stats misgivings

2021-05-03 Thread vignesh C
On Tue, May 4, 2021 at 9:48 AM Masahiko Sawada wrote: > > On Mon, May 3, 2021 at 10:21 PM Amit Kapila wrote: > > > > On Mon, May 3, 2021 at 5:48 PM Masahiko Sawada > > wrote: > > > > > > On Mon, May 3, 2021 at 2:27 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Apr 29, 2021 at 10:37 AM

Re: Replication slot stats misgivings

2021-05-03 Thread Masahiko Sawada
On Mon, May 3, 2021 at 10:21 PM Amit Kapila wrote: > > On Mon, May 3, 2021 at 5:48 PM Masahiko Sawada wrote: > > > > On Mon, May 3, 2021 at 2:27 PM Amit Kapila wrote: > > > > > > On Thu, Apr 29, 2021 at 10:37 AM Amit Kapila > > > wrote: > > > > > > > > On Wed, Apr 28, 2021 at 7:43 PM Masahiko

Re: Replication slot stats misgivings

2021-05-03 Thread Amit Kapila
On Mon, May 3, 2021 at 5:48 PM Masahiko Sawada wrote: > > On Mon, May 3, 2021 at 2:27 PM Amit Kapila wrote: > > > > On Thu, Apr 29, 2021 at 10:37 AM Amit Kapila > > wrote: > > > > > > On Wed, Apr 28, 2021 at 7:43 PM Masahiko Sawada > > > wrote: > > > > > > > > On Wed, Apr 28, 2021 at 3:25 PM

Re: Replication slot stats misgivings

2021-05-03 Thread Masahiko Sawada
On Mon, May 3, 2021 at 2:29 PM Amit Kapila wrote: > > On Fri, Apr 30, 2021 at 1:47 PM Amit Kapila wrote: > > > > LGTM. I have slightly edited the comments in the attached. I'll push > > this early next week unless there are more comments. > > > > Pushed. Thank you! Regards, -- Masahiko

Re: Replication slot stats misgivings

2021-05-03 Thread Masahiko Sawada
On Mon, May 3, 2021 at 2:27 PM Amit Kapila wrote: > > On Thu, Apr 29, 2021 at 10:37 AM Amit Kapila wrote: > > > > On Wed, Apr 28, 2021 at 7:43 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Apr 28, 2021 at 3:25 PM Amit Kapila > > > wrote: > > > > > > > > > > > I am not sure if any of

Re: Replication slot stats misgivings

2021-05-02 Thread Amit Kapila
On Fri, Apr 30, 2021 at 1:47 PM Amit Kapila wrote: > > LGTM. I have slightly edited the comments in the attached. I'll push > this early next week unless there are more comments. > Pushed. -- With Regards, Amit Kapila.

Re: Replication slot stats misgivings

2021-05-02 Thread Amit Kapila
On Thu, Apr 29, 2021 at 10:37 AM Amit Kapila wrote: > > On Wed, Apr 28, 2021 at 7:43 PM Masahiko Sawada wrote: > > > > On Wed, Apr 28, 2021 at 3:25 PM Amit Kapila wrote: > > > > > > > > I am not sure if any of these alternatives are a good idea. What do > > > you think? Do you have any other

Re: Replication slot stats misgivings

2021-04-30 Thread Amit Kapila
On Fri, Apr 30, 2021 at 5:55 AM Masahiko Sawada wrote: > > After more thought, it seems to me that we should use txn->size here > regardless of the top transaction or subtransaction since we're > iterating changes associated with a transaction that is either the top > transaction or a

Re: Replication slot stats misgivings

2021-04-30 Thread vignesh C
On Fri, Apr 30, 2021 at 5:55 AM Masahiko Sawada wrote: > > On Thu, Apr 29, 2021 at 9:44 PM vignesh C wrote: > > > > > > > > On Thu, Apr 29, 2021 at 3:06 PM Amit Kapila wrote: > > > > > > On Wed, Apr 28, 2021 at 5:01 PM Amit Kapila > > > wrote: > > > > > > > > On Wed, Apr 28, 2021 at 4:51 PM

Re: Replication slot stats misgivings

2021-04-29 Thread Amit Kapila
On Thu, Apr 29, 2021 at 12:07 PM Amit Kapila wrote: > > On Thu, Apr 29, 2021 at 11:14 AM Masahiko Sawada > wrote: > > > > > > > > How about doing both of the above suggestions? Alternatively, we can > > > wait for both 'drop' and 'create' message to be delivered but that > > > might be

Re: Replication slot stats misgivings

2021-04-29 Thread Masahiko Sawada
On Thu, Apr 29, 2021 at 9:44 PM vignesh C wrote: > > > > On Thu, Apr 29, 2021 at 3:06 PM Amit Kapila wrote: > > > > On Wed, Apr 28, 2021 at 5:01 PM Amit Kapila wrote: > > > > > > On Wed, Apr 28, 2021 at 4:51 PM Masahiko Sawada > > > wrote: > > > > > > > > On Wed, Apr 28, 2021 at 6:39 PM Amit

Re: Replication slot stats misgivings

2021-04-29 Thread vignesh C
On Thu, Apr 29, 2021 at 3:06 PM Amit Kapila wrote: > > On Wed, Apr 28, 2021 at 5:01 PM Amit Kapila wrote: > > > > On Wed, Apr 28, 2021 at 4:51 PM Masahiko Sawada wrote: > > > > > > On Wed, Apr 28, 2021 at 6:39 PM Amit Kapila wrote: > > > > @@ -1369,7 +1369,7 @@

Re: Replication slot stats misgivings

2021-04-29 Thread Amit Kapila
On Wed, Apr 28, 2021 at 5:01 PM Amit Kapila wrote: > > On Wed, Apr 28, 2021 at 4:51 PM Masahiko Sawada wrote: > > > > On Wed, Apr 28, 2021 at 6:39 PM Amit Kapila wrote: > > @@ -1369,7 +1369,7 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, > ReorderBufferIterTXNState *state) > * Update the

Re: Replication slot stats misgivings

2021-04-29 Thread vignesh C
On Thu, Apr 29, 2021 at 11:14 AM Masahiko Sawada wrote: > > On Thu, Apr 29, 2021 at 11:55 AM Amit Kapila wrote: > > > > On Thu, Apr 29, 2021 at 4:58 AM Masahiko Sawada > > wrote: > > > > > > On Thu, Apr 29, 2021 at 5:41 AM Tom Lane wrote: > > > > > > > > It seems that the test case added by

Re: Replication slot stats misgivings

2021-04-29 Thread Amit Kapila
On Thu, Apr 29, 2021 at 11:14 AM Masahiko Sawada wrote: > > > > > How about doing both of the above suggestions? Alternatively, we can > > wait for both 'drop' and 'create' message to be delivered but that > > might be overkill. > > Agreed. Attached the patch doing both things. > Thanks, the

Re: Replication slot stats misgivings

2021-04-28 Thread Masahiko Sawada
On Thu, Apr 29, 2021 at 11:55 AM Amit Kapila wrote: > > On Thu, Apr 29, 2021 at 4:58 AM Masahiko Sawada wrote: > > > > On Thu, Apr 29, 2021 at 5:41 AM Tom Lane wrote: > > > > > > It seems that the test case added by f5fc2f5b2 is still a bit > > > unstable, even after c64dcc7fe: > > > > Hmm, I

Re: Replication slot stats misgivings

2021-04-28 Thread Amit Kapila
On Wed, Apr 28, 2021 at 7:43 PM Masahiko Sawada wrote: > > On Wed, Apr 28, 2021 at 3:25 PM Amit Kapila wrote: > > > > > I am not sure if any of these alternatives are a good idea. What do > > you think? Do you have any other ideas for this? > > I've been considering some ideas but don't come up

Re: Replication slot stats misgivings

2021-04-28 Thread Amit Kapila
On Thu, Apr 29, 2021 at 8:50 AM Tom Lane wrote: > > Amit Kapila writes: > > This is the first test and inserts just one small record, so how it > > can lead to spill of data. Do you mean to say that may be some > > background process has written some transaction which leads to a spill > > of

Re: Replication slot stats misgivings

2021-04-28 Thread Andres Freund
On 2021-04-28 23:20:00 -0400, Tom Lane wrote: > (At least not before the fabled stats collector rewrite, which may well > introduce some entirely new set of failure modes.) FWIW, I added a function that forces a flush there. That can be done synchronously and the underlying functionality needs to

Re: Replication slot stats misgivings

2021-04-28 Thread Tom Lane
Amit Kapila writes: > This is the first test and inserts just one small record, so how it > can lead to spill of data. Do you mean to say that may be some > background process has written some transaction which leads to a spill > of data? autovacuum, say? > Yeah, something like this could

Re: Replication slot stats misgivings

2021-04-28 Thread Amit Kapila
On Thu, Apr 29, 2021 at 4:58 AM Masahiko Sawada wrote: > > On Thu, Apr 29, 2021 at 5:41 AM Tom Lane wrote: > > > > It seems that the test case added by f5fc2f5b2 is still a bit > > unstable, even after c64dcc7fe: > > Hmm, I don't see the exact cause yet but there are two possibilities: > some

Re: Replication slot stats misgivings

2021-04-28 Thread Masahiko Sawada
On Thu, Apr 29, 2021 at 5:41 AM Tom Lane wrote: > > It seems that the test case added by f5fc2f5b2 is still a bit > unstable, even after c64dcc7fe: Hmm, I don't see the exact cause yet but there are two possibilities: some transactions were really spilled, and it showed the old stats due to

Re: Replication slot stats misgivings

2021-04-28 Thread Tom Lane
It seems that the test case added by f5fc2f5b2 is still a bit unstable, even after c64dcc7fe: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=peripatus=2021-04-23%2006%3A20%3A12 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=peripatus=2021-04-24%2018%3A20%3A10

Re: Replication slot stats misgivings

2021-04-28 Thread Masahiko Sawada
On Wed, Apr 28, 2021 at 3:25 PM Amit Kapila wrote: > > On Wed, Apr 28, 2021 at 9:37 AM Masahiko Sawada wrote: > > > > On Wed, Apr 28, 2021 at 12:29 PM Amit Kapila > > wrote: > > > > > > On Tue, Apr 27, 2021 at 11:02 AM vignesh C wrote: > > > > > > > > On Tue, Apr 27, 2021 at 9:48 AM vignesh C

Re: Replication slot stats misgivings

2021-04-28 Thread Amit Kapila
On Wed, Apr 28, 2021 at 4:51 PM Masahiko Sawada wrote: > > On Wed, Apr 28, 2021 at 6:39 PM Amit Kapila wrote: > > > > > I think we can fix it by keeping track of total_size in toptxn as we > > are doing for the streaming case in ReorderBufferChangeMemoryUpdate. > > We can probably do it for

Re: Replication slot stats misgivings

2021-04-28 Thread Masahiko Sawada
On Wed, Apr 28, 2021 at 6:39 PM Amit Kapila wrote: > > On Wed, Apr 28, 2021 at 12:49 PM Masahiko Sawada > wrote: > > > > > > BTW regarding the commit f5fc2f5b23 that added total_txns and > > total_bytes, we add the reorder buffer size (i.g., rb->size) to > > rb->totalBytes but I think we should

Re: Replication slot stats misgivings

2021-04-28 Thread Amit Kapila
On Wed, Apr 28, 2021 at 12:49 PM Masahiko Sawada wrote: > > > BTW regarding the commit f5fc2f5b23 that added total_txns and > total_bytes, we add the reorder buffer size (i.g., rb->size) to > rb->totalBytes but I think we should use the transaction size (i.g., > txn->size) instead: > You are

Re: Replication slot stats misgivings

2021-04-28 Thread Masahiko Sawada
On Fri, Apr 16, 2021 at 2:58 PM Amit Kapila wrote: > > On Thu, Apr 15, 2021 at 4:35 PM Masahiko Sawada wrote: > > > > Thank you for the update! The patch looks good to me. > > BTW regarding the commit f5fc2f5b23 that added total_txns and total_bytes, we add the reorder buffer size (i.g.,

Re: Replication slot stats misgivings

2021-04-28 Thread Amit Kapila
On Wed, Apr 28, 2021 at 9:37 AM Masahiko Sawada wrote: > > On Wed, Apr 28, 2021 at 12:29 PM Amit Kapila wrote: > > > > On Tue, Apr 27, 2021 at 11:02 AM vignesh C wrote: > > > > > > On Tue, Apr 27, 2021 at 9:48 AM vignesh C wrote: > > > > > > > > > > Attached patch has the changes to update

Re: Replication slot stats misgivings

2021-04-27 Thread vignesh C
On Wed, Apr 28, 2021 at 9:37 AM Masahiko Sawada wrote: > > On Wed, Apr 28, 2021 at 12:29 PM Amit Kapila wrote: > > > > On Tue, Apr 27, 2021 at 11:02 AM vignesh C wrote: > > > > > > On Tue, Apr 27, 2021 at 9:48 AM vignesh C wrote: > > > > > > > > > > Attached patch has the changes to update

Re: Replication slot stats misgivings

2021-04-27 Thread Masahiko Sawada
On Wed, Apr 28, 2021 at 12:29 PM Amit Kapila wrote: > > On Tue, Apr 27, 2021 at 11:02 AM vignesh C wrote: > > > > On Tue, Apr 27, 2021 at 9:48 AM vignesh C wrote: > > > > > > > Attached patch has the changes to update statistics during > > spill/stream which prevents the statistics from being

Re: Replication slot stats misgivings

2021-04-27 Thread vignesh C
On Wed, Apr 28, 2021 at 8:59 AM Amit Kapila wrote: > > On Tue, Apr 27, 2021 at 11:02 AM vignesh C wrote: > > > > On Tue, Apr 27, 2021 at 9:48 AM vignesh C wrote: > > > > > > > Attached patch has the changes to update statistics during > > spill/stream which prevents the statistics from being

Re: Replication slot stats misgivings

2021-04-27 Thread Amit Kapila
On Tue, Apr 27, 2021 at 11:02 AM vignesh C wrote: > > On Tue, Apr 27, 2021 at 9:48 AM vignesh C wrote: > > > > Attached patch has the changes to update statistics during > spill/stream which prevents the statistics from being lost during > interrupt. > void

Re: Replication slot stats misgivings

2021-04-27 Thread vignesh C
On Wed, Apr 28, 2021 at 8:28 AM Amit Kapila wrote: > > On Tue, Apr 27, 2021 at 8:28 PM Masahiko Sawada wrote: > > > > On Tue, Apr 27, 2021 at 11:29 PM Amit Kapila > > wrote: > > > > > > On Tue, Apr 27, 2021 at 5:40 PM Amit Kapila > > > wrote: > > > > > > > > > > I am not sure if the timeout

Re: Replication slot stats misgivings

2021-04-27 Thread Amit Kapila
On Tue, Apr 27, 2021 at 8:28 PM Masahiko Sawada wrote: > > On Tue, Apr 27, 2021 at 11:29 PM Amit Kapila wrote: > > > > On Tue, Apr 27, 2021 at 5:40 PM Amit Kapila wrote: > > > > > > > I am not sure if the timeout happened because the machine is slow or > > is it in any way related to code. I am

Re: Replication slot stats misgivings

2021-04-27 Thread Masahiko Sawada
On Tue, Apr 27, 2021 at 11:29 PM Amit Kapila wrote: > > On Tue, Apr 27, 2021 at 5:40 PM Amit Kapila wrote: > > > > On Tue, Apr 27, 2021 at 8:58 AM Masahiko Sawada > > wrote: > > > > I have pushed this patch and seeing one buildfarm failure: > >

Re: Replication slot stats misgivings

2021-04-27 Thread Amit Kapila
On Tue, Apr 27, 2021 at 5:40 PM Amit Kapila wrote: > > On Tue, Apr 27, 2021 at 8:58 AM Masahiko Sawada wrote: > > I have pushed this patch and seeing one buildfarm failure: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mandrill=2021-04-27%2009%3A23%3A14 > > starting permutation:

Re: Replication slot stats misgivings

2021-04-27 Thread Amit Kapila
On Tue, Apr 27, 2021 at 8:58 AM Masahiko Sawada wrote: > > On Tue, Apr 27, 2021 at 11:45 AM Amit Kapila wrote: > > > > > > Sawada-San, I would like to go ahead with your > > "Use-HTAB-for-replication-slot-statistics" unless you think otherwise? > > I agree that it's better to use the stats

Re: Replication slot stats misgivings

2021-04-26 Thread vignesh C
On Tue, Apr 27, 2021 at 9:48 AM vignesh C wrote: > > On Tue, Apr 27, 2021 at 9:43 AM Amit Kapila wrote: > > > > On Tue, Apr 27, 2021 at 9:17 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Apr 27, 2021 at 11:31 AM vignesh C wrote: > > > > > > > > > > And I think there is > > > > > > also a

Re: Replication slot stats misgivings

2021-04-26 Thread Masahiko Sawada
On Tue, Apr 27, 2021 at 1:18 PM vignesh C wrote: > > On Tue, Apr 27, 2021 at 9:43 AM Amit Kapila wrote: > > > > On Tue, Apr 27, 2021 at 9:17 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Apr 27, 2021 at 11:31 AM vignesh C wrote: > > > > > > > > > > And I think there is > > > > > > also a

Re: Replication slot stats misgivings

2021-04-26 Thread vignesh C
On Tue, Apr 27, 2021 at 9:43 AM Amit Kapila wrote: > > On Tue, Apr 27, 2021 at 9:17 AM Masahiko Sawada wrote: > > > > On Tue, Apr 27, 2021 at 11:31 AM vignesh C wrote: > > > > > > > > And I think there is > > > > > also a risk to increase shared memory when we want to add other > > > > >

Re: Replication slot stats misgivings

2021-04-26 Thread Amit Kapila
On Tue, Apr 27, 2021 at 9:17 AM Masahiko Sawada wrote: > > On Tue, Apr 27, 2021 at 11:31 AM vignesh C wrote: > > > > > > And I think there is > > > > also a risk to increase shared memory when we want to add other > > > > statistics in the future. > > > > > > > > > > Yeah, so do you think it is

Re: Replication slot stats misgivings

2021-04-26 Thread Masahiko Sawada
On Tue, Apr 27, 2021 at 11:31 AM vignesh C wrote: > > On Mon, Apr 26, 2021 at 8:42 AM Amit Kapila wrote: > > > > On Mon, Apr 26, 2021 at 8:01 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Apr 23, 2021 at 6:15 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, Apr 19, 2021 at 4:28 PM

Re: Replication slot stats misgivings

2021-04-26 Thread Masahiko Sawada
On Tue, Apr 27, 2021 at 11:45 AM Amit Kapila wrote: > > On Tue, Apr 27, 2021 at 8:01 AM vignesh C wrote: > > > > On Mon, Apr 26, 2021 at 8:42 AM Amit Kapila wrote: > > > > > > On Mon, Apr 26, 2021 at 8:01 AM Masahiko Sawada > > > wrote: > > > > > > > > On Fri, Apr 23, 2021 at 6:15 PM Amit

Re: Replication slot stats misgivings

2021-04-26 Thread Amit Kapila
On Tue, Apr 27, 2021 at 8:01 AM vignesh C wrote: > > On Mon, Apr 26, 2021 at 8:42 AM Amit Kapila wrote: > > > > On Mon, Apr 26, 2021 at 8:01 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Apr 23, 2021 at 6:15 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, Apr 19, 2021 at 4:28 PM

Re: Replication slot stats misgivings

2021-04-26 Thread vignesh C
On Mon, Apr 26, 2021 at 8:42 AM Amit Kapila wrote: > > On Mon, Apr 26, 2021 at 8:01 AM Masahiko Sawada wrote: > > > > On Fri, Apr 23, 2021 at 6:15 PM Amit Kapila wrote: > > > > > > On Mon, Apr 19, 2021 at 4:28 PM vignesh C wrote: > > > > > > > > I have made the changes to update the

Re: Replication slot stats misgivings

2021-04-25 Thread Amit Kapila
On Mon, Apr 26, 2021 at 8:01 AM Masahiko Sawada wrote: > > On Fri, Apr 23, 2021 at 6:15 PM Amit Kapila wrote: > > > > On Mon, Apr 19, 2021 at 4:28 PM vignesh C wrote: > > > > > > I have made the changes to update the replication statistics at > > > replication slot release. Please find the

Re: Replication slot stats misgivings

2021-04-25 Thread Masahiko Sawada
On Fri, Apr 23, 2021 at 6:15 PM Amit Kapila wrote: > > On Mon, Apr 19, 2021 at 4:28 PM vignesh C wrote: > > > > I have made the changes to update the replication statistics at > > replication slot release. Please find the patch attached for the same. > > Thoughts? > > > > Thanks, the changes

Re: Replication slot stats misgivings

2021-04-23 Thread Amit Kapila
On Mon, Apr 19, 2021 at 4:28 PM vignesh C wrote: > > I have made the changes to update the replication statistics at > replication slot release. Please find the patch attached for the same. > Thoughts? > Thanks, the changes look mostly good to me. The slot stats need to be initialized in

Re: Replication slot stats misgivings

2021-04-22 Thread Amit Kapila
On Thu, Apr 22, 2021 at 1:02 PM Masahiko Sawada wrote: > Thanks, it looks good to me now. I'll review/test some more before committing but at this stage, I would like to know from Andres or others whether they see any problem with this approach to fixing a few of the problems reported in this

Re: Replication slot stats misgivings

2021-04-22 Thread Masahiko Sawada
On Thu, Apr 22, 2021 at 3:03 PM Amit Kapila wrote: > > On Thu, Apr 22, 2021 at 10:39 AM Masahiko Sawada > wrote: > > > > On Thu, Apr 22, 2021 at 1:50 PM Amit Kapila wrote: > > > > > > On Thu, Apr 22, 2021 at 8:26 AM Masahiko Sawada > > > wrote: > > > > > > > > > 2. > > > + if

Re: Replication slot stats misgivings

2021-04-22 Thread Amit Kapila
On Thu, Apr 22, 2021 at 10:39 AM Masahiko Sawada wrote: > > On Thu, Apr 22, 2021 at 1:50 PM Amit Kapila wrote: > > > > On Thu, Apr 22, 2021 at 8:26 AM Masahiko Sawada > > wrote: > > > > > > 2. > > + if (replSlotStatHash != NULL) > > + (void) hash_search(replSlotStatHash, > > +(void *)

Re: Replication slot stats misgivings

2021-04-21 Thread Masahiko Sawada
On Thu, Apr 22, 2021 at 1:50 PM Amit Kapila wrote: > > On Thu, Apr 22, 2021 at 8:26 AM Masahiko Sawada wrote: > > > > Few comments: > 1. > I think we want stats collector to not use pgStatLocalContext unless > it has read the stats file similar to other cases. So probably, we > should allocate

Re: Replication slot stats misgivings

2021-04-21 Thread Amit Kapila
On Thu, Apr 22, 2021 at 8:26 AM Masahiko Sawada wrote: > Few comments: 1. I think we want stats collector to not use pgStatLocalContext unless it has read the stats file similar to other cases. So probably, we should allocate it in pgStatLocalContext when we read 'R' message in

Re: Replication slot stats misgivings

2021-04-21 Thread Dilip Kumar
On Thu, Apr 22, 2021 at 7:52 AM Masahiko Sawada wrote: > > On Wed, Apr 21, 2021 at 4:44 PM Dilip Kumar wrote: > > > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada > > wrote: > > > > > > > > I've attached the patch. In addition to the test Vignesh prepared, I > > > added one test for the

Re: Replication slot stats misgivings

2021-04-21 Thread Masahiko Sawada
On Wed, Apr 21, 2021 at 7:11 PM Amit Kapila wrote: > > On Wed, Apr 21, 2021 at 3:39 PM Masahiko Sawada wrote: > > > > > > > > The test is not waiting for a new slot creation message to reach the > > > stats collector. So, if the old slot data still exists in the file and > > > now when we read

Re: Replication slot stats misgivings

2021-04-21 Thread Masahiko Sawada
On Wed, Apr 21, 2021 at 4:44 PM Dilip Kumar wrote: > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada wrote: > > > > > I've attached the patch. In addition to the test Vignesh prepared, I > > added one test for the message for creating a slot that checks if the > > statistics are initialized

Re: Replication slot stats misgivings

2021-04-21 Thread Amit Kapila
On Wed, Apr 21, 2021 at 3:39 PM Masahiko Sawada wrote: > > > > > The test is not waiting for a new slot creation message to reach the > > stats collector. So, if the old slot data still exists in the file and > > now when we read stats via backend, then won't there exists a chance > > that old

Re: Replication slot stats misgivings

2021-04-21 Thread Masahiko Sawada
On Wed, Apr 21, 2021 at 6:36 PM Amit Kapila wrote: > > On Wed, Apr 21, 2021 at 2:46 PM Masahiko Sawada wrote: > > > > On Wed, Apr 21, 2021 at 3:09 PM Amit Kapila wrote: > > > > > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada > > > wrote: > > > > > > > > > > > > I've attached the patch. In

Re: Replication slot stats misgivings

2021-04-21 Thread Masahiko Sawada
On Wed, Apr 21, 2021 at 6:20 PM Amit Kapila wrote: > > On Wed, Apr 21, 2021 at 2:37 PM Masahiko Sawada wrote: > > > > On Wed, Apr 21, 2021 at 12:50 PM Amit Kapila > > wrote: > > > > > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada > > > wrote: > > > > > > > > > > I have one question: > >

Re: Replication slot stats misgivings

2021-04-21 Thread Amit Kapila
On Wed, Apr 21, 2021 at 2:46 PM Masahiko Sawada wrote: > > On Wed, Apr 21, 2021 at 3:09 PM Amit Kapila wrote: > > > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada > > wrote: > > > > > > > > > I've attached the patch. In addition to the test Vignesh prepared, I > > > added one test for the

Re: Replication slot stats misgivings

2021-04-21 Thread Amit Kapila
On Wed, Apr 21, 2021 at 2:37 PM Masahiko Sawada wrote: > > On Wed, Apr 21, 2021 at 12:50 PM Amit Kapila wrote: > > > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada > > wrote: > > > > > > > I have one question: > > > > + /* > > + * Create the replication slot stats hash table if we don't

Re: Replication slot stats misgivings

2021-04-21 Thread Masahiko Sawada
On Wed, Apr 21, 2021 at 3:09 PM Amit Kapila wrote: > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada wrote: > > > > > > I've attached the patch. In addition to the test Vignesh prepared, I > > added one test for the message for creating a slot that checks if the > > statistics are initialized

Re: Replication slot stats misgivings

2021-04-21 Thread Masahiko Sawada
On Wed, Apr 21, 2021 at 12:50 PM Amit Kapila wrote: > > On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada wrote: > > > > I have one question: > > + /* > + * Create the replication slot stats hash table if we don't have > + * it already. > + */ > + if (replSlotStats == NULL) > { > - if

Re: Replication slot stats misgivings

2021-04-21 Thread Dilip Kumar
On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada wrote: > > I've attached the patch. In addition to the test Vignesh prepared, I > added one test for the message for creating a slot that checks if the > statistics are initialized after re-creating the same name slot. > Please review it. Overall

Re: Replication slot stats misgivings

2021-04-21 Thread Amit Kapila
On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada wrote: > > > I've attached the patch. In addition to the test Vignesh prepared, I > added one test for the message for creating a slot that checks if the > statistics are initialized after re-creating the same name slot. > I am not sure how much

Re: Replication slot stats misgivings

2021-04-20 Thread vignesh C
On Wed, Apr 21, 2021 at 9:47 AM Amit Kapila wrote: > > On Wed, Apr 21, 2021 at 9:39 AM vignesh C wrote: > > > > I feel we can change CATALOG_VERSION_NO so that we will get this error > > "The database cluster was initialized with CATALOG_VERSION_NO > > 2021X, but the server was compiled with

Re: Replication slot stats misgivings

2021-04-20 Thread Amit Kapila
On Wed, Apr 21, 2021 at 9:39 AM vignesh C wrote: > > I feel we can change CATALOG_VERSION_NO so that we will get this error > "The database cluster was initialized with CATALOG_VERSION_NO > 2021X, but the server was compiled with CATALOG_VERSION_NO > 2021X." which will prevent the above

Re: Replication slot stats misgivings

2021-04-20 Thread vignesh C
On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada wrote: > > On Tue, Apr 20, 2021 at 7:22 PM vignesh C wrote: > > > > On Tue, Apr 20, 2021 at 9:08 AM Masahiko Sawada > > wrote: > > > > > > On Mon, Apr 19, 2021 at 4:48 PM Masahiko Sawada > > > wrote: > > > > > > > > On Mon, Apr 19, 2021 at 2:14

Re: Replication slot stats misgivings

2021-04-20 Thread Amit Kapila
On Tue, Apr 20, 2021 at 7:54 PM Masahiko Sawada wrote: > I have one question: + /* + * Create the replication slot stats hash table if we don't have + * it already. + */ + if (replSlotStats == NULL) { - if (namestrcmp([i].slotname, name) == 0) - return i; /* found */ + HASHCTL hash_ctl; + +

Re: Replication slot stats misgivings

2021-04-20 Thread Masahiko Sawada
On Tue, Apr 20, 2021 at 7:22 PM vignesh C wrote: > > On Tue, Apr 20, 2021 at 9:08 AM Masahiko Sawada wrote: > > > > On Mon, Apr 19, 2021 at 4:48 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Apr 19, 2021 at 2:14 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, Apr 19, 2021 at 9:00 AM

Re: Replication slot stats misgivings

2021-04-20 Thread Masahiko Sawada
On Tue, Apr 20, 2021 at 6:59 PM Amit Kapila wrote: > > On Tue, Apr 20, 2021 at 9:08 AM Masahiko Sawada wrote: > > > > I've attached the new version patch that fixed the compilation error > > reported off-line by Amit. > > > > I was thinking about whether we can someway avoid the below risk: > In

  1   2   3   >