Re: Catalog version access

2022-01-30 Thread Michael Paquier
On Tue, Jan 25, 2022 at 01:12:32PM +0900, Michael Paquier wrote: > Once you remove the requirement of a running server, we have basically > what has been recently implemented with postgres -C for > runtime-computed GUCs, because we already go through a read of the > control file to be able to print

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-30 Thread Bharath Rupireddy
On Mon, Jan 31, 2022 at 12:47 PM Michael Paquier wrote: > > On Mon, Jan 31, 2022 at 01:54:16PM +0800, Julien Rouhaud wrote: > > For now we have some room, but we will likely keep consuming bytes in that > > file > > for more critical features and it's almost certain that at one point we will > >

Re: row filtering for logical replication

2022-01-30 Thread Amit Kapila
On Mon, Jan 31, 2022 at 7:27 AM houzj.f...@fujitsu.com wrote: > > On Monday, January 31, 2022 8:53 AM Peter Smith wrote: > > > > PSA v73*. > > > > (A rebase was needed due to recent changes in tab-complete.c. > > Otherwise, v73* is the same as v72*). > > Thanks for the rebase. > Attach the V74 pa

Re: Suppressing useless wakeups in walreceiver

2022-01-30 Thread Kyotaro Horiguchi
At Fri, 28 Jan 2022 22:41:32 +1300, Thomas Munro wrote in > On Fri, Jan 28, 2022 at 8:26 PM Kyotaro Horiguchi > wrote: > > At Thu, 27 Jan 2022 23:50:04 +1300, Thomas Munro > > wrote in > The reason why I put the timeout computation into a function is > because there are about 3 places you nee

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2022-01-30 Thread Michael Paquier
On Sat, Jan 29, 2022 at 08:07:23PM +0530, Bharath Rupireddy wrote: > If the server crashes in end-of-recovery, in the follow-up startup, > the server has to start all the recovery right? In that case, > DB_IN_{ARCHIVE, CRASH}_RECOVERY would represent the correct state to > the user, not the DB_SHUT

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-30 Thread Michael Paquier
On Mon, Jan 31, 2022 at 01:54:16PM +0800, Julien Rouhaud wrote: > For now we have some room, but we will likely keep consuming bytes in that > file > for more critical features and it's almost certain that at one point we will > regret wasting 2 bytes for that. Agreed to drop the patch. That's o

Re: row filtering for logical replication

2022-01-30 Thread Greg Nancarrow
On Mon, Jan 31, 2022 at 12:57 PM houzj.f...@fujitsu.com wrote: > > Attach the V74 patch set which did the following changes: > Hi, I tested psql and pg_dump after application of this patch, from the following perspectives: - "\dRp+" and "\d " (added by the patch, for PostgreSQL 15) show row filt

Re: Column Filtering in Logical Replication

2022-01-30 Thread Peter Eisentraut
On 12.01.22 01:41, Alvaro Herrera wrote: I discovered a big hole in this, which is that ALTER PUBLICATION SET (publish='insert,update') can add UPDATE publishing to a publication that was only publishing INSERTs. It's easy to implement a fix: in AlterPublicationOptions, scan the list of tables a

Re: make MaxBackends available in _PG_init

2022-01-30 Thread Michael Paquier
On Sat, Jan 29, 2022 at 02:24:24PM +0900, Michael Paquier wrote: > Yeah, it would be good to know the scope before defining the limits > of what could be done. Another thing may be the interactions with > session_preload_libraries and local_preload_libraries. Worth noting that I have marked marke

Re: Error "initial slot snapshot too large" in create replication slot

2022-01-30 Thread Dilip Kumar
On Mon, Jan 31, 2022 at 11:50 AM Kyotaro Horiguchi wrote: > > SnapBUildInitialSnapshot tries to store XIDS of both top and sub > transactions into snapshot->xip array but the array is easily > overflowed and CREATE_REPLICATOIN_SLOT command ends with an error. > > To fix this, this patch is doing t

Re: Add header support to text format and matching feature

2022-01-30 Thread Peter Eisentraut
On 30.01.22 23:56, Rémi Lapeyre wrote: I notice in the 0002 patch that there is no test case for the error "wrong header for column \"%s\": got \"%s\"", which I think is really the core functionality of this patch. So please add that. I added a test for it in this new version of the patch.

Re: Error "initial slot snapshot too large" in create replication slot

2022-01-30 Thread Kyotaro Horiguchi
At Mon, 17 Jan 2022 09:27:14 +0530, Dilip Kumar wrote in > On Wed, Jan 12, 2022 at 4:09 PM Julien Rouhaud wrote: > > The cfbot reports that this patch doesn't compile: > > https://cirrus-ci.com/task/564273490432?logs=build > > > > [03:01:24.477] snapbuild.c: In function ‘SnapBuildInitialSna

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-30 Thread Julien Rouhaud
Hi, On Mon, Jan 31, 2022 at 10:58:31AM +0530, Bharath Rupireddy wrote: > > The size of ControlFileData is 296 bytes currently and the sector > limit is of 512 bytes (PG_CONTROL_MAX_SAFE_SIZE), if we feel that this > extra 2 bytes of checkpoint flags isn't worth storing in the control > file, I'm

Re: Latest LLVM breaks our code again

2022-01-30 Thread Thomas Munro
On Mon, Jan 31, 2022 at 3:38 PM Tom Lane wrote: > Speaking of buildfarm breakage, seawasp has been failing for the > past several days. It looks like bleeding-edge LLVM has again > changed some APIs we depend on. First failure is here: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=

Re: Spelling change in LLVM 14 API

2022-01-30 Thread Thomas Munro
On Tue, Oct 26, 2021 at 2:21 PM Thomas Munro wrote: > Here's one like that. The previous message "Track LLVM 14 API > changes" didn't seem too scalable so I added date and commit ID. seawasp finally caught up with these LLVM changes and turned red. I retested the patch against this week's LLVM

Re: Error "initial slot snapshot too large" in create replication slot

2022-01-30 Thread Kyotaro Horiguchi
At Tue, 2 Nov 2021 16:40:39 +0530, Dilip Kumar wrote in > On Tue, Oct 19, 2021 at 2:25 PM Dilip Kumar wrote: > > > > On Tue, Oct 12, 2021 at 11:30 AM Dilip Kumar wrote: > > > > > > On Tue, Oct 12, 2021 at 10:35 AM Kyotaro Horiguchi > > > wrote: > > > > > > > > At Tue, 12 Oct 2021 13:59:59 +090

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-30 Thread Bharath Rupireddy
On Mon, Jan 31, 2022 at 9:10 AM Julien Rouhaud wrote: > > Hi, > > On Mon, Jan 31, 2022 at 11:10:45AM +0900, Kyotaro Horiguchi wrote: > > > > This means pg_controldata need to translate the flags into human-readable > > text but, to be clear, I still don't think its usefull in the control > > data.

Re: GUC flags

2022-01-30 Thread Michael Paquier
On Sat, Jan 29, 2022 at 06:18:50PM -0600, Justin Pryzby wrote: > "The most meaningful ones are included" doesn't seem to add anything. > Maybe it'd be useful to say "(Only the most useful flags are exposed)" Yes, I have used something like that. > I think the description is wrong, or just copied

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-30 Thread Bharath Rupireddy
On Thu, Jan 27, 2022 at 6:31 AM Nathan Bossart wrote: > > I spent some time thinking about the right way to proceed here, and I came > up with the attached patches. The first patch just adds error checking for > various lstat() calls in the replication code. If lstat() fails, then it > probably

Re: Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-30 Thread David Rowley
On Fri, 28 Jan 2022 at 09:20, Peter Geoghegan wrote: > Both VACUUM and ANALYZE update pg_class.reltuples. But this code seems > to assume that it's only something that VACUUM can ever do. Like Justin I'm also not quite following what the problem is here. pg_class.reltuples is only used to estimat

Re: pgsql: Server-side gzip compression.

2022-01-30 Thread David Rowley
On Fri, 28 Jan 2022 at 08:44, Robert Haas wrote: > OK. I'm still surprised, but it is what it is. I've committed this now. Thanks > FWIW, I would have been fine with you just committing this change. That's good to know, thanks for mentioning it. FWIW, I just held back as I wasn't 100% sure on

Re: Printing backtrace of postgres processes

2022-01-30 Thread vignesh C
On Sat, Jan 29, 2022 at 8:06 AM vignesh C wrote: > > On Fri, Jan 28, 2022 at 1:54 PM Bharath Rupireddy > wrote: > > > > On Thu, Jan 27, 2022 at 10:45 AM vignesh C wrote: > > > > > > On Wed, Jan 26, 2022 at 11:07 AM Bharath Rupireddy > > > wrote: > > > > > > > > On Tue, Jan 25, 2022 at 12:00 PM

Re: DELETE CASCADE

2022-01-30 Thread Julien Rouhaud
Hi, On Tue, Jan 25, 2022 at 10:26:53PM +0800, Julien Rouhaud wrote: > > It's been almost 4 months since your last email, and almost 2 weeks since the > notice that this patch doesn't apply anymore. Without update in the next > couple of days this patch will be closed as Returned with Feedback pe

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-30 Thread Julien Rouhaud
Hi, On Mon, Jan 31, 2022 at 11:10:45AM +0900, Kyotaro Horiguchi wrote: > > This means pg_controldata need to translate the flags into human-readable > text but, to be clear, I still don't think its usefull in the control > data. I've been saying that since my first email, I also don't see any sc

Re: [BUG]Update Toast data failure in logical replication

2022-01-30 Thread Dilip Kumar
On Sat, Jan 29, 2022 at 3:57 PM Amit Kapila wrote: > > On Fri, Jan 28, 2022 at 12:16 PM Dilip Kumar wrote: > > > + /* > + * If it's a whole-tuple reference, say "not equal". It's not really > + * worth supporting this case, since it could only succeed after a > + * no-op update, which is hardly

Re: row filtering for logical replication

2022-01-30 Thread Greg Nancarrow
On Mon, Jan 31, 2022 at 1:12 PM houzj.f...@fujitsu.com wrote: > > > > + /* > > > +* We need this map to avoid relying on ReorderBufferChangeType > > enums > > > +* having specific values. > > > +*/ > > > + static int map_changetype_pubaction[] = { > > > + [REORDER_BUFFER_

Latest LLVM breaks our code again

2022-01-30 Thread Tom Lane
Speaking of buildfarm breakage, seawasp has been failing for the past several days. It looks like bleeding-edge LLVM has again changed some APIs we depend on. First failure is here: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2022-01-28%2000%3A17%3A48

RE: row filtering for logical replication

2022-01-30 Thread houzj.f...@fujitsu.com
On Saturday, January 29, 2022 8:31 AM Andres Freund wrote: > > Hi, > > Are there any recent performance evaluations of the overhead of row filters? I > think it'd be good to get some numbers comparing: Thanks for looking at the patch! Will test it. > 1) $workload with master > 2) $workload wit

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-30 Thread Kyotaro Horiguchi
At Sat, 29 Jan 2022 00:10:19 +0800, Julien Rouhaud wrote in > On Fri, Jan 28, 2022 at 08:21:52PM +0530, Bharath Rupireddy wrote: > > > Also, you still didn't fix the possible flag upgrade issue. > > Unless I'm missing something that's an issue that you still haven't addressed > or explained why

Re: Support tab completion for upper character inputs in psql

2022-01-30 Thread Tom Lane
=?utf-8?Q?Dagfinn_Ilmari_Manns=C3=A5ker?= writes: > First, as noted in the test, it doesn't preserve the case of the input > for keywords appended to the query result. This is easily fixed by > using `pg_strdup_keyword_case()`, per the first attached patch. I thought about that, and intentionall

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-30 Thread Kyotaro Horiguchi
At Fri, 28 Jan 2022 13:49:19 +0530, Bharath Rupireddy wrote in > > - proallargtypes => > > '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}', > > + proallargtypes => > > '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,

Re: Support tab completion for upper character inputs in psql

2022-01-30 Thread Dagfinn Ilmari Mannsåker
Tom Lane writes: > "tanghy.f...@fujitsu.com" writes: >> Thanks for your V16 patch, I tested it. >> The results LGTM. > > Pushed, thanks for looking. I wasn't following this thread, but I noticed a few small potential improvements when I saw the commit. First, as noted in the test, it doesn't

Re: CREATEROLE and role ownership hierarchies

2022-01-30 Thread Mark Dilger
> On Jan 30, 2022, at 2:38 PM, Michael Banck wrote: > > Hi, Your review is greatly appreciated! >> The attached WIP patch attempts to solve most of the CREATEROLE I'm mostly looking for whether the general approach in this Work In Progress patch is acceptable, so I was a bit sloppy with wh

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-30 Thread Andres Freund
Hi, On 2022-01-29 13:47:13 -0800, Andres Freund wrote: > Here's a version of the patch only creating the event once. Needs a small bit > of comment polishing, but otherwise I think it's sane? Ah, it needs a bit more. I was not cleaning up the event at the exit of ReceiveXlogStream(). For pg_baseb

Re: plperl on windows

2022-01-30 Thread Andres Freund
Hi, On 2022-01-30 15:14:32 -0800, Noah Misch wrote: > Last I looked (~2017), EDB distributed an MSVC-built Perl as the designated > Perl to use with https://www.postgresql.org/download/windows/ plperl. Ah, interesting. I didn't find a perl binary in the archive offered, and I didn't immediately f

Re: plperl on windows

2022-01-30 Thread Noah Misch
On Sun, Jan 30, 2022 at 02:16:59PM -0800, Andres Freund wrote: > Specifically where USE_THREAD_SAFE_LOCALE is defined for msvc. Which explains > why the same perl build ends up with different definitions for > PerlInterpreter, depending on headers getting compiled with gcc or > msvc. > > Seems pre

Re: Add header support to text format and matching feature

2022-01-30 Thread Rémi Lapeyre
> On 28 Jan 2022, at 09:57, Peter Eisentraut > wrote: > > On 31.12.21 18:36, Rémi Lapeyre wrote: >> Here’s an updated version of the patch that takes into account the changes >> in d1029bb5a2. The actual code is the same as v10 which was already marked >> as ready for committer. > > I have c

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-30 Thread Andres Freund
Hi, On 2022-01-30 16:51:12 +0100, Magnus Hagander wrote: > On Sat, Jan 29, 2022 at 10:47 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-01-29 12:44:22 -0800, Andres Freund wrote: > > > On 2022-01-17 10:06:56 -0800, Andres Freund wrote: > > > > Yes, that's what I was suggesting. I wasn't thin

Re: Postgresql Windows build and modern perl (>=5.28)

2022-01-30 Thread Andres Freund
Hi, On 2021-10-04 23:08:25 +0100, Dagfinn Ilmari Mannsåker wrote: > Victor Wagner writes: > > > Attached patch makes use of this function if PERL_VERSION >= 28. > > It makes plperl compile with ActiveStatePerl 5.28 and StrawberryPerl > > 5.30.2.1. > > I have no opinion on the substantive conte

Re: CREATEROLE and role ownership hierarchies

2022-01-30 Thread Michael Banck
Hi, On Sat, Jan 29, 2022 at 09:58:38PM -0800, Mark Dilger wrote: > > On Jan 25, 2022, at 12:44 PM, Stephen Frost wrote: > > I agree that CREATEROLE is overpowered and that the goal of this should > > be to provide a way for roles to be created and dropped that doesn't > > give the user who has th

Re: plperl on windows

2022-01-30 Thread Andres Freund
Hi, On 2022-01-30 12:56:16 -0800, Andres Freund wrote: > The gcc version has a Ilc_numeric_mutex_depth that the msvc version > doesn't. The relevant part of intrpvar.h: > > PERLVAR(I, in_utf8_turkic_locale, bool) > #if defined(USE_ITHREADS) && ! defined(USE_THREAD_SAFE_LOCALE) > PERLVARI(I, lc_nu

Re: drop tablespace failed when location contains .. on win32

2022-01-30 Thread Tom Lane
Michael Paquier writes: > In order to make the tests cheap, there is no need to have a separate > module in src/test/modules/ as your patch 0002 is doing. Instead, you > should move the C code of your SQL function test_canonicalize_path() > to src/test/regress/regress.c, then add some tests in >

Re: plperl on windows

2022-01-30 Thread Andres Freund
Hi, On 2021-10-04 14:38:16 -0700, Andres Freund wrote: > 3) When building against strawberry perl 5.32.1.1 I see errors when loading >plperl The error is: loadable library and perl binaries are mismatched (got handshake key 12800080, needed 12900080) A bunch of research led

Re: Support tab completion for upper character inputs in psql

2022-01-30 Thread Tom Lane
"tanghy.f...@fujitsu.com" writes: > Thanks for your V16 patch, I tested it. > The results LGTM. Pushed, thanks for looking. regards, tom lane

Re: BufferAlloc: don't take two simultaneous locks

2022-01-30 Thread Michail Nikolaev
Hello, Yura. Test results look promising. But it seems like the naming and dynahash API change is a little confusing. 1) I think it is better to split the main part and atomic nentries optimization into separate commits. 2) Also, it would be nice to also fix hash_update_hash_key bug :) 3) Do we r

Re: Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-30 Thread Justin Pryzby
On Thu, Jan 27, 2022 at 01:59:38PM -0800, Peter Geoghegan wrote: > On Thu, Jan 27, 2022 at 12:20 PM Peter Geoghegan wrote: > > Both VACUUM and ANALYZE update pg_class.reltuples. But this code seems > > to assume that it's only something that VACUUM can ever do. Why > > wouldn't we expect a plain A

Re: [PATCH] nodeindexscan with reorder memory leak

2022-01-30 Thread Tom Lane
Aliaksandr Kalenik writes: > I was investigating a leak reported in the PostGIS issues tracker [1] which > led me to the Postgres side where the problem really is. The leak is > reproducible with query from original ticket [1]: > ... > The leak is only noticeable when index scan with reorder happe

Re: [PATCH] nodeindexscan with reorder memory leak

2022-01-30 Thread Tom Lane
Aliaksandr Kalenik writes: > The leak is only noticeable when index scan with reorder happens as part of > subquery plan which is explained by the fact that heap tuples cloned in > reorderqueue_push are not freed during flush of reorder queue in > ExecReScanIndex. Hmm ... I see from the code cove

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-30 Thread Magnus Hagander
On Sat, Jan 29, 2022 at 10:47 PM Andres Freund wrote: > > Hi, > > On 2022-01-29 12:44:22 -0800, Andres Freund wrote: > > On 2022-01-17 10:06:56 -0800, Andres Freund wrote: > > > Yes, that's what I was suggesting. I wasn't thinking of using a static > > > var, > > > but putting it in StreamCtl. No

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-30 Thread Magnus Hagander
On Sat, Jan 29, 2022 at 9:44 PM Andres Freund wrote: > > On 2022-01-17 10:06:56 -0800, Andres Freund wrote: > > Yes, that's what I was suggesting. I wasn't thinking of using a static var, > > but putting it in StreamCtl. Note that what pgwin32_waitforsinglesocket() > > is doing doesn't protect aga

Re: Remove extra includes of "access/xloginsert.h" when "access/xlog.h" is included

2022-01-30 Thread Alvaro Herrera
On 2022-Jan-30, Bharath Rupireddy wrote: > Here's the v3 patch removing xloginsert.h from xlog.h and adding > xloginsert.h in the required files. Pushed, but I added xloginsert.h to 9 additional files that needed it to avoid compiler warnings. Thanks! -- Álvaro Herrera Valdivia, C

Re: Remove extra includes of "access/xloginsert.h" when "access/xlog.h" is included

2022-01-30 Thread Julien Rouhaud
Hi, On Sun, Jan 30, 2022 at 06:52:48PM +0530, Bharath Rupireddy wrote: > > Here's the v3 patch removing xloginsert.h from xlog.h and adding > xloginsert.h in the required files. +1, this approach is better. In general it's better to increase the number of include lines rather than having a few

Re: Remove extra includes of "access/xloginsert.h" when "access/xlog.h" is included

2022-01-30 Thread Bharath Rupireddy
On Sun, Jan 30, 2022 at 1:07 AM Alvaro Herrera wrote: > > On 2022-Jan-29, Bharath Rupireddy wrote: > > > Removing the xloginsert.h in xlog.h would need us to add xloginsert.h > > in more areas. > > Sure. > > > And also, it might break any non-core extensions that > > includes just xlog.h and gets

RE: logical replication empty transactions

2022-01-30 Thread osumi.takami...@fujitsu.com
On Thursday, January 27, 2022 9:57 PM Ajin Cherian wrote: Hi, thanks for your patch update. > On Wed, Jan 26, 2022 at 8:33 PM osumi.takami...@fujitsu.com > wrote: > > > > On Tuesday, January 11, 2022 6:43 PM Ajin Cherian > wrote: > > (3) Is this patch's reponsibility to intialize the data in >