Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Dilip Kumar
On Thu, Dec 30, 2021 at 12:36 PM SATYANARAYANA NARLAPURAM < satyanarlapu...@gmail.com> wrote: > >> Yeah, I think that would make sense, even though we will be allowing a >> new backend to get connected insert WAL, and get committed but after that, >> it will be throttled. However, if the number

Re: Per-table storage parameters for TableAM/IndexAM extensions

2021-12-29 Thread Dilip Kumar
On Wed, Dec 29, 2021 at 10:38 PM Sadhuprasad Patro wrote: > Hi, > > Currently all the storage options for a table are very much specific > to the heap but a different AM might need some user defined AM > specific parameters to help tune the AM. So here is a patch which > provides an AM level

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread SATYANARAYANA NARLAPURAM
On Wed, Dec 29, 2021 at 10:38 PM Dilip Kumar wrote: > On Thu, Dec 30, 2021 at 1:09 AM Andres Freund wrote: > >> Hi, >> >> On 2021-12-29 11:34:53 -0800, SATYANARAYANA NARLAPURAM wrote: >> > On Wed, Dec 29, 2021 at 11:31 AM Andres Freund >> wrote: >> > Andres, thanks for the comments. Agreed on

RE: Confused comment about drop replica identity index

2021-12-29 Thread houzj.f...@fujitsu.com
On Tues, Dec 21, 2021 8:47 AM Michael Paquier wrote: > On Mon, Dec 20, 2021 at 11:57:32AM -0300, Euler Taveira wrote: > > What do you think about the attached patch? It forbids the DROP INDEX. > > We might add a detail message but I didn't in this patch. > > Yeah. I'd agree about doing

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Dilip Kumar
On Thu, Dec 30, 2021 at 1:09 AM Andres Freund wrote: > Hi, > > On 2021-12-29 11:34:53 -0800, SATYANARAYANA NARLAPURAM wrote: > > On Wed, Dec 29, 2021 at 11:31 AM Andres Freund > wrote: > > Andres, thanks for the comments. Agreed on this based on the previous > > discussions on this thread.

Re: Tests "with" and "alter_table" suffer from name clash

2021-12-29 Thread Tom Lane
Thomas Munro writes: > In fact only REL_10_STABLE had the problem, because commit 2cf8c7aa > already fixed the other instance in later branches. I'd entirely > forgotten that earlier discussion, which apparently didn't quite go > far enough. So I only needed to push the with.sql change. Done.

Re: Tests "with" and "alter_table" suffer from name clash

2021-12-29 Thread Thomas Munro
On Thu, Dec 30, 2021 at 3:27 PM Tom Lane wrote: > Looks reasonable. We really should avoid using such common > names for short-lived tables in any case --- it's an invitation > to trouble. So I'd vote for changing the other use of "test", too. In fact only REL_10_STABLE had the problem,

Re: Tests "with" and "alter_table" suffer from name clash

2021-12-29 Thread Tom Lane
Thomas Munro writes: > With unlucky scheduling you can get a failure like this: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly=2021-12-22%2010%3A51%3A32 > Suggested fix attached. Looks reasonable. We really should avoid using such common names for short-lived tables in any

Tests "with" and "alter_table" suffer from name clash

2021-12-29 Thread Thomas Munro
Hi, With unlucky scheduling you can get a failure like this: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly=2021-12-22%2010%3A51%3A32 Suggested fix attached. From 3991f040e9c9afc4d7cfd4980b5f27f4113dbd1f Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 30 Dec 2021

Re: PublicationActions - use bit flags.

2021-12-29 Thread Tom Lane
Peter Smith writes: > On Thu, Dec 30, 2021 at 3:30 AM Justin Pryzby wrote: >> + if (pubform->pubinsert) pub->pubactions |= PUBACTION_INSERT; >> This is usually written like: >> pub->pubactions |= (pubform->pubinsert ? PUBACTION_INSERT : 0) > Thanks for the info, I've modified those

Re: PublicationActions - use bit flags.

2021-12-29 Thread Peter Smith
On Thu, Dec 30, 2021 at 3:30 AM Justin Pryzby wrote: > > On Mon, Dec 20, 2021 at 11:18:41AM +1100, Peter Smith wrote: > > For some reason the current HEAD PublicationActions is a struct of > > boolean representing combinations of the 4 different "publication > > actions". > > > > I felt it is

Re: Column Filtering in Logical Replication

2021-12-29 Thread Alvaro Herrera
On 2021-Dec-28, Alvaro Herrera wrote: > There are still some XXX comments. The one that bothers me most is the > lack of an implementation that allows changing the column list in a > publication without having to remove the table from the publication > first. OK, I made some progress on this

Re: UNIQUE null treatment option

2021-12-29 Thread Zhihong Yu
Hi, boolisunique; + boolnulls_not_distinct; } BTSpool; Looking at the other fields in BTSpool, there is no underscore in field name. I think the new field can be named nullsdistinct. This way, the double negative is avoided. Similar comment for new fields in BTShared and

SELECT documentation

2021-12-29 Thread Joel Jacobson
Hi, The Examples section in the documentation for the SELECT command [1] only contains a single example on how to join two tables, which is written in SQL-89 style: SELECT f.title, f.did, d.name, f.date_prod, f.kind FROM distributors d, films f WHERE f.did = d.did I think it's good to

Re: Strange path from pgarch_readyXlog()

2021-12-29 Thread Tom Lane
"Bossart, Nathan" writes: > On 12/29/21, 1:04 PM, "Tom Lane" wrote: >> While we're here, I wonder if we ought to get rid of the static-ness of >> these arrays. I realize that they're only eating a few kB, but they're >> doing so in every postgres process, when they'll only be used in the >>

Re: Logging replication state changes

2021-12-29 Thread SATYANARAYANA NARLAPURAM
On Wed, Dec 29, 2021 at 2:04 PM Tom Lane wrote: > SATYANARAYANA NARLAPURAM writes: > > I noticed that below critical replication state changes are DEBUG1 level > > logged. Any concern about changing the below two messages log level to > LOG? > > Why? These seem like perfectly routine messages.

Re: Adding CI to our tree

2021-12-29 Thread Daniel Gustafsson
> On 29 Dec 2021, at 21:17, Andres Freund wrote: > On 2021-12-20 11:21:05 -0800, Andres Freund wrote: >> Attached is v4 of the CI patch. > > I'd like to push this - any objections? It's not disruptive to anything but > cfbot, so we can incrementally improve it further. No objection, I'm +1 on

Re: Logging replication state changes

2021-12-29 Thread Tom Lane
SATYANARAYANA NARLAPURAM writes: > I noticed that below critical replication state changes are DEBUG1 level > logged. Any concern about changing the below two messages log level to LOG? Why? These seem like perfectly routine messages. regards, tom lane

Logging replication state changes

2021-12-29 Thread SATYANARAYANA NARLAPURAM
Hi hackers, I noticed that below critical replication state changes are DEBUG1 level logged. Any concern about changing the below two messages log level to LOG? If this is too verbose, we can introduce a new GUC, log_replication_state_changes that logs the replication state changes when enabled

Re: Strange path from pgarch_readyXlog()

2021-12-29 Thread Tom Lane
"Bossart, Nathan" writes: > I bet this was a simple mistake in beb4e9b. > -static char arch_filenames[NUM_FILES_PER_DIRECTORY_SCAN][MAX_XFN_CHARS]; > +static char arch_filenames[NUM_FILES_PER_DIRECTORY_SCAN][MAX_XFN_CHARS + 1]; Hm, yeah, that looks like a pretty obvious bug. While we're here,

Re: Strange path from pgarch_readyXlog()

2021-12-29 Thread Bossart, Nathan
On 12/29/21, 12:22 PM, "Thomas Munro" wrote: > Isn't this a corrupted pathname? > > 2021-12-29 03:39:55.708 CST [79851:1] WARNING: removal of orphan > archive status file > "pg_wal/archive_status/00010003.0028.backup00010004.ready" > failed too many times,

Re: Add Boolean node

2021-12-29 Thread Andres Freund
Hi, On 2021-12-27 10:02:14 +0100, Peter Eisentraut wrote: > This patch adds a new node type Boolean, to go alongside the "value" nodes > Integer, Float, String, etc. This seems appropriate given that Boolean > values are a fundamental part of the system and are used a lot. > > Before, SQL-level

Re: Add Boolean node

2021-12-29 Thread Tom Lane
Andres Freund writes: > If we go around changing all these places, it might be worth to also change > Integer to be a int64 instead of an int. Meh ... that would have some non-obvious consequences, I think, at least if you tried to make the grammar make use of the extra width (it'd change the

Re: Add Boolean node

2021-12-29 Thread Andres Freund
On 2021-12-27 09:53:32 -0500, Tom Lane wrote: > Didn't really read the patch in any detail, but I did have one idea: > I think that the different things-that-used-to-be-Value-nodes ought to > use different field names, say ival, rval, bval, sval not just "val". > That makes it more likely that

Strange path from pgarch_readyXlog()

2021-12-29 Thread Thomas Munro
Hi, Isn't this a corrupted pathname? 2021-12-29 03:39:55.708 CST [79851:1] WARNING: removal of orphan archive status file "pg_wal/archive_status/00010003.0028.backup00010004.ready" failed too many times, will try again later

Re: Adding CI to our tree

2021-12-29 Thread Andres Freund
Hi, On 2021-12-20 11:21:05 -0800, Andres Freund wrote: > Attached is v4 of the CI patch. I'd like to push this - any objections? It's not disruptive to anything but cfbot, so we can incrementally improve it further. I'll try to sync pushing with Thomas, so that he can adjust cfbot to not add

Re: WIP: WAL prefetch (another approach)

2021-12-29 Thread Andres Freund
Hi, On 2021-12-29 17:29:52 +1300, Thomas Munro wrote: > > FWIW I don't think we include updates to typedefs.list in patches. > > Seems pretty harmless? And useful to keep around in development > branches because I like to pgindent stuff... I think it's even helpful. As long as it's done with a

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Andres Freund
Hi, On 2021-12-29 11:34:53 -0800, SATYANARAYANA NARLAPURAM wrote: > On Wed, Dec 29, 2021 at 11:31 AM Andres Freund wrote: > Andres, thanks for the comments. Agreed on this based on the previous > discussions on this thread. Could you please share your thoughts on adding > it after

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread SATYANARAYANA NARLAPURAM
On Wed, Dec 29, 2021 at 11:31 AM Andres Freund wrote: > Hi, > > On 2021-12-27 16:40:28 -0800, SATYANARAYANA NARLAPURAM wrote: > > > Yet another problem is that if we are in XlogInsert() that means we are > > > holding the buffer locks on all the pages we have modified, so if we > add a > > >

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Andres Freund
Hi, On 2021-12-27 16:40:28 -0800, SATYANARAYANA NARLAPURAM wrote: > > Yet another problem is that if we are in XlogInsert() that means we are > > holding the buffer locks on all the pages we have modified, so if we add a > > hook at that level which can make it wait then we would also block any

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread SATYANARAYANA NARLAPURAM
On Wed, Dec 29, 2021 at 11:16 AM Stephen Frost wrote: > Greetings, > > On Wed, Dec 29, 2021 at 14:04 SATYANARAYANA NARLAPURAM < > satyanarlapu...@gmail.com> wrote: > >> Stephen, thank you! >> >> On Wed, Dec 29, 2021 at 5:46 AM Stephen Frost wrote: >> >>> Greetings, >>> >>> * SATYANARAYANA

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Stephen Frost
Greetings, On Wed, Dec 29, 2021 at 14:04 SATYANARAYANA NARLAPURAM < satyanarlapu...@gmail.com> wrote: > Stephen, thank you! > > On Wed, Dec 29, 2021 at 5:46 AM Stephen Frost wrote: > >> Greetings, >> >> * SATYANARAYANA NARLAPURAM (satyanarlapu...@gmail.com) wrote: >> > On Sat, Dec 25, 2021 at

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread SATYANARAYANA NARLAPURAM
Stephen, thank you! On Wed, Dec 29, 2021 at 5:46 AM Stephen Frost wrote: > Greetings, > > * SATYANARAYANA NARLAPURAM (satyanarlapu...@gmail.com) wrote: > > On Sat, Dec 25, 2021 at 9:25 PM Dilip Kumar > wrote: > > > On Sun, Dec 26, 2021 at 10:36 AM SATYANARAYANA NARLAPURAM < > > >

Re: Report checkpoint progress in server logs

2021-12-29 Thread SATYANARAYANA NARLAPURAM
Coincidentally, I was thinking about the same yesterday after tired of waiting for the checkpoint completion on a server. On Wed, Dec 29, 2021 at 7:41 AM Tom Lane wrote: > Magnus Hagander writes: > >> Therefore, reporting the checkpoint progress in the server logs, much > >> like [1], seems

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-29 Thread Justin Pryzby
http://cfbot.cputube.org/sami-imseih.html You should run "make check" and update rules.out. You should also use make check-world - usually something like: make check-world -j4 >check-world.out 2>&1 ; echo ret $? > indrelid: The relid of the index currently being vacuumed I think it should be

Per-table storage parameters for TableAM/IndexAM extensions

2021-12-29 Thread Sadhuprasad Patro
Hi, Currently all the storage options for a table are very much specific to the heap but a different AM might need some user defined AM specific parameters to help tune the AM. So here is a patch which provides an AM level routine so that instead of getting parameters validated using

Re: PublicationActions - use bit flags.

2021-12-29 Thread Justin Pryzby
On Mon, Dec 20, 2021 at 11:18:41AM +1100, Peter Smith wrote: > For some reason the current HEAD PublicationActions is a struct of > boolean representing combinations of the 4 different "publication > actions". > > I felt it is more natural to implement boolean flag combinations using > a bitmask

Re: Report checkpoint progress in server logs

2021-12-29 Thread Tom Lane
Magnus Hagander writes: >> Therefore, reporting the checkpoint progress in the server logs, much >> like [1], seems to be the best way IMO. > I find progress reporting in the logfile to generally be a terrible > way of doing things, and the fact that we do it for the startup > process is/should

Re: Foreign key joins revisited

2021-12-29 Thread Tom Lane
Peter Eisentraut writes: > In the 1990s, there were some SQL drafts that included syntax like > JOIN ... USING PRIMARY KEY | USING FOREIGN KEY | USING CONSTRAINT ... > AFAICT, these ideas just faded away because of other priorities, so if > someone wants to revive it, some work already exists.

Re: Foreign key joins revisited

2021-12-29 Thread Andrew Dunstan
On 12/28/21 15:10, Tom Lane wrote: > Vik Fearing writes: >> On 12/28/21 8:26 PM, Joel Jacobson wrote: >>> Can with think of some other suitable reserved keyword? >> I don't particularly like this whole idea anyway, but if we're going to >> have it, I would suggest >> JOIN ... USING KEY ...

Re: [PATCH] allow src/tools/msvc/*.bat files to be called from the root of the source tree

2021-12-29 Thread Andrew Dunstan
On 12/29/21 05:16, Anton Voloshin wrote: > Hello, > > currently, on Windows/MSVC, src\tools\msvc\*.bat files mostly require > being in that src\tools\msvc directory first. > > I suggest an obvious fix: [...] > This patch uses standard windows cmd's %~dp0 to get the complete path > (drive, "d",

Re: pg_archivecleanup - add the ability to detect, archive and delete the unneeded wal files on the primary

2021-12-29 Thread Bharath Rupireddy
On Wed, Dec 29, 2021 at 7:27 PM Stephen Frost wrote: > > On Thu, Dec 23, 2021, at 9:58 AM, Bharath Rupireddy wrote: > > > pg_archivecleanup currently takes a WAL file name as input to delete > > > the WAL files prior to it [1]. As suggested by Satya (cc-ed) in > > > pg_replslotdata thread [2],

Re: Report checkpoint progress in server logs

2021-12-29 Thread Magnus Hagander
On Wed, Dec 29, 2021 at 3:31 PM Bharath Rupireddy wrote: > > Hi, > > At times, some of the checkpoint operations such as removing old WAL > files, dealing with replication snapshot or mapping files etc. may > take a while during which the server doesn't emit any logs or > information, the only

Report checkpoint progress in server logs

2021-12-29 Thread Bharath Rupireddy
Hi, At times, some of the checkpoint operations such as removing old WAL files, dealing with replication snapshot or mapping files etc. may take a while during which the server doesn't emit any logs or information, the only logs emitted are LogCheckpointStart and LogCheckpointEnd. Many times this

Re: Proposal: More structured logging

2021-12-29 Thread Justin Pryzby
> Subject: [PATCH v3 2/3] Add test module for the new tag functionality. ... > +test_logging(PG_FUNCTION_ARGS) > +{ ... > + (errmsg("%s", message), > + ({ > + forboth(lk, keys, lv, values) > + { > + (errtag(lfirst(lk), "%s", (char *)

Re: pg_archivecleanup - add the ability to detect, archive and delete the unneeded wal files on the primary

2021-12-29 Thread Stephen Frost
Greetings, * Euler Taveira (eu...@eulerto.com) wrote: > On Thu, Dec 23, 2021, at 9:58 AM, Bharath Rupireddy wrote: > > pg_archivecleanup currently takes a WAL file name as input to delete > > the WAL files prior to it [1]. As suggested by Satya (cc-ed) in > > pg_replslotdata thread [2], can we

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Stephen Frost
Greetings, * SATYANARAYANA NARLAPURAM (satyanarlapu...@gmail.com) wrote: > On Sat, Dec 25, 2021 at 9:25 PM Dilip Kumar wrote: > > On Sun, Dec 26, 2021 at 10:36 AM SATYANARAYANA NARLAPURAM < > > satyanarlapu...@gmail.com> wrote: > >>> Actually all the WAL insertions are done under a critical

Re: Documenting when to retry on serialization failure

2021-12-29 Thread Simon Riggs
On Wed, 29 Dec 2021 at 03:30, Thomas Munro wrote: > > On Fri, Dec 10, 2021 at 1:43 AM Simon Riggs > wrote: > > "Applications using this level must be prepared to retry transactions > > due to serialization failures." > > ... > > "When an application receives this error message, it should abort

Re: generalized conveyor belt storage

2021-12-29 Thread Amul Sul
On Wed, Dec 15, 2021 at 9:04 PM Robert Haas wrote: > > On Wed, Dec 15, 2021 at 10:03 AM Matthias van de Meent > wrote: > [...] Thought patch is WIP, here are a few comments that I found while reading the patch and thought might help: + { + if (meta->cbm_oldest_index_segment == +

Re: Converting WAL to SQL

2021-12-29 Thread Fabrízio de Royes Mello
On Wed, 29 Dec 2021 at 03:18 rajesh singarapu wrote: > Hi Hackers, > > I am wondering if we have a mechanism to convert WAL records to SQL > statements. > > I am able to use logical decoders like wal2json or test_decoding for > converting WAL to readable format, but I am looking for a way to

Re: Logical replication timeout problem

2021-12-29 Thread Fabrice Chapuis
I put the instance with high level debug mode. I try to do some log interpretation: After having finished writing the modifications generated by the insert in the snap files, then these files are read (restored). One minute after this work starts, the worker process exit with an error code = 1. I

Re: automatically generating node support functions

2021-12-29 Thread Peter Eisentraut
On 12.10.21 15:52, Andrew Dunstan wrote: I haven't been through the whole thing, but I did notice this: the comment stripping code looks rather fragile. I think it would blow up if there were a continuation line not starting with  qr/\s*\*/. It's a lot simpler and more robust to do this if you

[PATCH] allow src/tools/msvc/*.bat files to be called from the root of the source tree

2021-12-29 Thread Anton Voloshin
Hello, currently, on Windows/MSVC, src\tools\msvc\*.bat files mostly require being in that src\tools\msvc directory first. I suggest an obvious fix: diff --git a/src/tools/msvc/build.bat b/src/tools/msvc/build.bat index 4001ac1d0d1..407b6559cfb 100755 --- a/src/tools/msvc/build.bat +++

Re: UNIQUE null treatment option

2021-12-29 Thread Peter Eisentraut
Here is a rebased version of this patch. On 27.08.21 14:38, Peter Eisentraut wrote: The SQL standard has been ambiguous about whether null values in unique constraints should be considered equal or not.  Different implementations have different behaviors.  In the SQL:202x draft, this has been

Re: Converting WAL to SQL

2021-12-29 Thread Peter Eisentraut
On 29.12.21 07:18, rajesh singarapu wrote: I am wondering if we have a mechanism to convert WAL records to SQL statements. I am able to use logical decoders like wal2json or test_decoding for converting WAL to readable format, but I am looking for a way to convert WAL to sql statements.

Re: Foreign key joins revisited

2021-12-29 Thread Peter Eisentraut
On 28.12.21 20:45, Vik Fearing wrote: I don't particularly like this whole idea anyway, but if we're going to have it, I would suggest JOIN ... USING KEY ... since USING currently requires a parenthesized list, that shouldn't create any ambiguity. In the 1990s, there were some SQL