Re: Logical replication timeout problem

2022-04-05 Thread Amit Kapila
On Wed, Apr 6, 2022 at 11:09 AM wangw.f...@fujitsu.com wrote: > > On Fri, Apr 1, 2022 at 12:09 AM Amit Kapila wrote: > > On Fri, Apr 1, 2022 at 8:28 AM Euler Taveira wrote: > > > > > > It seems I didn't make myself clear. The callbacks I'm referring to the > > > *_cb_wrapper functions. After

RE: Logical replication timeout problem

2022-04-05 Thread wangw.f...@fujitsu.com
On Fri, Apr 1, 2022 at 12:09 AM Amit Kapila wrote: > On Fri, Apr 1, 2022 at 8:28 AM Euler Taveira wrote: > > > > On Thu, Mar 31, 2022, at 11:27 PM, Amit Kapila wrote: > > > > This is exactly our initial analysis and we have tried a patch on > > these lines and it has a noticeable overhead. See

Re: API stability

2022-04-05 Thread Kyotaro Horiguchi
me> I'd like to do that. Let me see. At Tue, 5 Apr 2022 10:29:03 -0400, Robert Haas wrote in > On Tue, Apr 5, 2022 at 10:17 AM Tom Lane wrote: > > What I think you need to do is: > > > > 1. In the back branches, revert delayChkpt to its previous type and > > semantics. Squeeze a separate

Re: zero char is returned as space

2022-04-05 Thread Konstantin Izmailov
Tom, thank you very much! It makes sense now. K On Tue, Apr 5, 2022 at 10:08 PM Tom Lane wrote: > Konstantin Izmailov writes: > > could you help me understand if this is an intended behaviour, or I'm > > incorrectly querying a "char" field? > > We do not support '\0' as an element of a string

Perform streaming logical transactions by background workers and parallel apply

2022-04-05 Thread Amit Kapila
In this email, I would like to discuss allowing streaming logical transactions (large in-progress transactions) by background workers and parallel apply in general. The goal of this work is to improve the performance of the apply work in logical replication. Currently, for large transactions, the

Re: zero char is returned as space

2022-04-05 Thread Tom Lane
Konstantin Izmailov writes: > could you help me understand if this is an intended behaviour, or I'm > incorrectly querying a "char" field? We do not support '\0' as an element of a string value. You didn't show how you're trying to insert this value, but I suspect that Postgres saw it as an

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-04-05 Thread Jeff Davis
On Mon, 2022-04-04 at 09:15 +0530, Bharath Rupireddy wrote: > My intention is to return the overall undecoded WAL record [5] i.e. > the data starting from XLogReadRecord's output [6] till length > XLogRecGetTotalLen(xlogreader);. Please see [7], where Andres agreed > to have this function, I also

zero char is returned as space

2022-04-05 Thread Konstantin Izmailov
Hi, could you help me understand if this is an intended behaviour, or I'm incorrectly querying a "char" field? I have simple table with column declared as: c_tinyint char NOT NULL The column contains tiny integers in range 0-10. When I query the column from my app using libpq values 1-10 are

Re: Logical replication row filtering and TOAST

2022-04-05 Thread Amit Kapila
On Wed, Apr 6, 2022 at 7:21 AM Ajin Cherian wrote: > > On Tue, Apr 5, 2022 at 8:32 PM Amit Kapila wrote: > > > > > How about something like the attached? > > > > LGTM. > Pushed. -- With Regards, Amit Kapila.

Re: Skipping logical replication transactions on subscriber side

2022-04-05 Thread Amit Kapila
On Wed, Apr 6, 2022 at 9:25 AM Masahiko Sawada wrote: > > On Wed, Apr 6, 2022 at 12:21 PM Noah Misch wrote: > > Right. I've attached an updated patch. > Thanks, this looks good to me as well. Noah, would you like to commit it? -- With Regards, Amit Kapila.

Re: Skip partition tuple routing with constant partition key

2022-04-05 Thread Tom Lane
Amit Langote writes: > On Sun, Apr 3, 2022 at 10:31 PM Greg Stark wrote: >> Is this a problem with the patch or its tests? >> [18:14:20.798] Test Summary Report >> [18:14:20.798] --- >> [18:14:20.798] t/013_partition.pl (Wstat: 15360 Tests: 31 Failed: 0) > Hmm, make check-world

Re: Handle infinite recursion in logical replication setup

2022-04-05 Thread Amit Kapila
On Tue, Apr 5, 2022 at 7:06 PM Ashutosh Bapat wrote: > > On Tue, Apr 5, 2022 at 6:21 AM Peter Smith wrote: > > > > Below are some other name ideas. Maybe they are not improvements, but > > it might help other people to come up with something better. > > > > subscribe_publocal_only = true/false >

Re: Skipping logical replication transactions on subscriber side

2022-04-05 Thread Masahiko Sawada
On Wed, Apr 6, 2022 at 12:21 PM Noah Misch wrote: > > On Tue, Apr 05, 2022 at 04:41:28PM +0900, Masahiko Sawada wrote: > > On Tue, Apr 5, 2022 at 4:08 PM Noah Misch wrote: > > > On Tue, Apr 05, 2022 at 03:05:10PM +0900, Masahiko Sawada wrote: > > > > I've attached an updated patch. The patch

Re: Skip partition tuple routing with constant partition key

2022-04-05 Thread Amit Langote
On Sun, Apr 3, 2022 at 10:31 PM Greg Stark wrote: > Is this a problem with the patch or its tests? > > [18:14:20.798] # poll_query_until timed out executing this query: > [18:14:20.798] # SELECT count(1) = 0 FROM pg_subscription_rel WHERE > srsubstate NOT IN ('r', 's'); > [18:14:20.798] #

Re: shared-memory based stats collector - v69

2022-04-05 Thread David G. Johnston
On Tue, Apr 5, 2022 at 8:14 PM Andres Freund wrote: > > On 2022-04-05 20:00:50 -0700, David G. Johnston wrote: > > * Statistics are loaded from the filesystem during startup (by the startup > * process), unless preceded by a crash, in which case all stats are > * discarded. They are written

Re: SQL/JSON: JSON_TABLE

2022-04-05 Thread Tom Lane
Andres Freund writes: > The new jsonb_sqljson test is, on my machine, the slowest test in the main > regression tests: > ... > Any chance the slowness isn't required slowness? In general, there's been a serious bump in the buildfarm cycle time in the last month --- for example, on my admittedly

Re: shared-memory based stats collector - v70

2022-04-05 Thread Andres Freund
Hi, On 2022-04-05 23:11:07 -0400, Greg Stark wrote: > I've never tried to review a 24-patch series before. It's kind of > intimidating Is there a good place to start to get a good idea of > the most important changes? It was more at some point :). And believe me, I find this whole project

Re: Skipping logical replication transactions on subscriber side

2022-04-05 Thread Noah Misch
On Tue, Apr 05, 2022 at 04:41:28PM +0900, Masahiko Sawada wrote: > On Tue, Apr 5, 2022 at 4:08 PM Noah Misch wrote: > > On Tue, Apr 05, 2022 at 03:05:10PM +0900, Masahiko Sawada wrote: > > > I've attached an updated patch. The patch includes a regression test > > > to detect the new violation as

Re: shared-memory based stats collector - v70

2022-04-05 Thread David G. Johnston
On Tue, Apr 5, 2022 at 8:11 PM Greg Stark wrote: > I've never tried to review a 24-patch series before. It's kind of > intimidating Is there a good place to start to get a good idea of > the most important changes? > It isn't as bad as the number makes it sound - I just used "git am" to

Re: shared-memory based stats collector - v69

2022-04-05 Thread Andres Freund
Hi, On 2022-04-05 20:00:50 -0700, David G. Johnston wrote: > On Tue, Apr 5, 2022 at 4:16 PM Andres Freund wrote: > > On 2022-04-05 14:43:49 -0700, David G. Johnston wrote: > > > On Tue, Apr 5, 2022 at 2:23 PM Andres Freund wrote: > > > > I guess I should add a paragraph about snapshots / fetch

Re: shared-memory based stats collector - v70

2022-04-05 Thread Greg Stark
I've never tried to review a 24-patch series before. It's kind of intimidating Is there a good place to start to get a good idea of the most important changes?

Re: shared-memory based stats collector - v70

2022-04-05 Thread David G. Johnston
On Tue, Apr 5, 2022 at 8:00 PM Andres Freund wrote: > > Here comes v70: > > I think this is basically ready, minus a a few comment adjustments here and > there. Unless somebody protests I'm planning to start pushing things > tomorrow > morning. > > Nothing I've come across, given my area of

Re: shared-memory based stats collector - v69

2022-04-05 Thread David G. Johnston
On Tue, Apr 5, 2022 at 4:16 PM Andres Freund wrote: > On 2022-04-05 14:43:49 -0700, David G. Johnston wrote: > > On Tue, Apr 5, 2022 at 2:23 PM Andres Freund wrote: > > > > > > I guess I should add a paragraph about snapshots / fetch consistency. > > > > > > > I apparently confused/combined the

Re: [PATCH] pg_stat_toast

2022-04-05 Thread Andres Freund
Hi, On 2022-04-06 00:08:13 +0200, Gunnar "Nick" Bluth wrote: > AFAICT, Andres' work is more about the structure (e.g. > 13619598f1080d7923454634a2570ca1bc0f2fec). Or I've missed something...? That was just the prep work... I'm about to send slightly further polished version, but here's the

Re: SQL/JSON: JSON_TABLE

2022-04-05 Thread Andres Freund
Hi, On 2022-03-27 16:53:57 -0400, Andrew Dunstan wrote: > I'm therefore going to commit this series The new jsonb_sqljson test is, on my machine, the slowest test in the main regression tests: 4639 ms jsonb_sqljson 2401 ms btree_index 2166 ms stats_ext 2027 ms alter_table 1616 ms triggers 1498

Re: psql - add SHOW_ALL_RESULTS option - pg_regress output

2022-04-05 Thread Andres Freund
Hi, On 2022-04-04 23:32:50 +0200, Peter Eisentraut wrote: > This has been committed. It's somewhat annoying that made pg_regress even more verbose than before: == removing existing temp instance== == creating temporary instance

Re: Logical replication row filtering and TOAST

2022-04-05 Thread Ajin Cherian
On Tue, Apr 5, 2022 at 8:32 PM Amit Kapila wrote: > > How about something like the attached? > LGTM. regards, Ajin Cherian Fujitsu Australia

Re: API stability

2022-04-05 Thread Kyotaro Horiguchi
At Tue, 5 Apr 2022 10:01:56 -0400, Robert Haas wrote in > I think as the person who committed that patch I'm on the hook to fix > this if nobody else would like to do it, but let me ask whether > Kyotaro Horiguchi would like to propose a patch, since the original > patch did, and/or whether you

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Andy Fan
On Wed, Apr 6, 2022 at 12:41 AM Robert Haas wrote: > On Tue, Apr 5, 2022 at 10:10 AM Andy Fan wrote: > >> > I wanted to have a WAL record spanning multiple WAL files of size, say > >> > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table > >> > would help here. Please let me

Re: [COMMITTERS] pgsql: Allow time delayed standbys and recovery

2022-04-05 Thread Thom Brown
On Wed, 6 Apr 2022 at 01:42, Thom Brown wrote: > > On Tue, 5 Apr 2022 at 16:02, Robert Haas wrote: > > > > On Tue, Apr 5, 2022 at 10:58 AM Magnus Hagander wrote: > > >> Makes sense. I will do this soon if nobody objects. > > >> > > >> I'm mildly uncomfortable with the phrase "WAL records

Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors

2022-04-05 Thread Tatsuo Ishii
>> I would suggest to reorder the last chunk to: >> >>... retried retries failures serfail dlfail >> >> because I intend to add connection failures handling at some point, >> and it would make more sense to add the corresponding count at the end >> with other fails. > > Ok, I have adjusted

Re: Race condition in server-crash testing

2022-04-05 Thread Tom Lane
Andres Freund writes: > On 2022-04-04 00:50:27 -0400, Tom Lane wrote: >> It's hard to be totally sure, but I think what happened is that >> gaur hit the in-hindsight-obvious race condition in this code: >> we managed to execute a successful iteration of poll_query_until >> before the postmaster

Re: [COMMITTERS] pgsql: Allow time delayed standbys and recovery

2022-04-05 Thread Thom Brown
On Tue, 5 Apr 2022 at 16:02, Robert Haas wrote: > > On Tue, Apr 5, 2022 at 10:58 AM Magnus Hagander wrote: > >> Makes sense. I will do this soon if nobody objects. > >> > >> I'm mildly uncomfortable with the phrase "WAL records generated over > >> the delay period" because it seems a bit

Re: shared-memory based stats collector - v69

2022-04-05 Thread Andres Freund
On 2022-04-05 14:43:49 -0700, David G. Johnston wrote: > On Tue, Apr 5, 2022 at 2:23 PM Andres Freund wrote: > > > > > On 2022-04-05 13:51:12 -0700, David G. Johnston wrote: > > > > >, but rather add to the shared queue > > > > Queue? Maybe you mean the hashtable? > > > > Queue implemented by a

Re: Atomic rename feature for Windows.

2022-04-05 Thread Victor Spirin
Hi Updated patch: we use the posix semantic features in Windows build 17763 and up. We found an issue with this feature on Windows Server 2016 without updates (Windows 1607 Build 14393) Victor Spirin Postgres Professional:http://www.postgrespro.com The Russian Postgres Company 05.07.2021

Re: MERGE bug report

2022-04-05 Thread Zhihong Yu
On Tue, Apr 5, 2022 at 3:35 PM Zhihong Yu wrote: > > > On Tue, Apr 5, 2022 at 3:18 PM Joe Wildish wrote: > >> Hello Hackers, >> >> Reporting a bug with the new MERGE statement. Tested against >> 75edb919613ee835e7680e40137e494c7856bcf9. >> >> psql output as follows: >> >> ... >>

Re: MERGE bug report

2022-04-05 Thread Zhihong Yu
On Tue, Apr 5, 2022 at 3:18 PM Joe Wildish wrote: > Hello Hackers, > > Reporting a bug with the new MERGE statement. Tested against > 75edb919613ee835e7680e40137e494c7856bcf9. > > psql output as follows: > > ... > psql:merge.sql:33: ERROR: variable not found in subplan target lists > ROLLBACK >

Re: should vacuum's first heap pass be read-only?

2022-04-05 Thread Peter Geoghegan
On Tue, Apr 5, 2022 at 2:53 PM Robert Haas wrote: > On Tue, Apr 5, 2022 at 4:30 PM Peter Geoghegan wrote: > > On Tue, Apr 5, 2022 at 1:10 PM Robert Haas wrote: > > > I had assumed that this would not be the case, because if the page is > > > being accessed by the workload, it can be pruned -

MERGE bug report

2022-04-05 Thread Joe Wildish
Hello Hackers, Reporting a bug with the new MERGE statement. Tested against 75edb919613ee835e7680e40137e494c7856bcf9. psql output as follows: ... psql:merge.sql:33: ERROR: variable not found in subplan target lists ROLLBACK [local] joe@joe=# \errverbose ERROR: XX000: variable not found in

Re: [PATCH] pg_stat_toast

2022-04-05 Thread Gunnar "Nick" Bluth
Am 05.04.22 um 18:17 schrieb Robert Haas: > On Thu, Mar 31, 2022 at 9:16 AM Gunnar "Nick" Bluth > wrote: >> That was meant to say "v10", sorry! > > Hi, Hi Robert, and thx for looking at this. > From my point of view, at least, it would be preferable if you'd stop > changing the subject line

Re: should vacuum's first heap pass be read-only?

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 4:30 PM Peter Geoghegan wrote: > On Tue, Apr 5, 2022 at 1:10 PM Robert Haas wrote: > > I had assumed that this would not be the case, because if the page is > > being accessed by the workload, it can be pruned - and probably frozen > > too, if we wanted to write code for

Re: shared-memory based stats collector - v69

2022-04-05 Thread David G. Johnston
On Tue, Apr 5, 2022 at 2:23 PM Andres Freund wrote: > > On 2022-04-05 13:51:12 -0700, David G. Johnston wrote: > > >, but rather add to the shared queue > > Queue? Maybe you mean the hashtable? > Queue implemented by a list...? Anyway, I think I mean this: /* * List of PgStat_EntryRefs with

Re: shared-memory based stats collector - v69

2022-04-05 Thread Andres Freund
Hi, On 2022-04-05 13:51:12 -0700, David G. Johnston wrote: > On Mon, Apr 4, 2022 at 8:05 PM Andres Freund wrote: > > > - added an architecture overview comment to the top of pgstat.c - not sure > > if > > it makes sense to anybody but me (and perhaps Horiguchi-san)? > > > > > I took a look at

Re: [PATCH] Add extra statistics to explain for Nested Loop

2022-04-05 Thread Greg Stark
This is not passing regression tests due to some details of the plan output - marking Waiting on Author: diff -w -U3 c:/cirrus/src/test/regress/expected/partition_prune.out c:/cirrus/src/test/recovery/tmp_check/results/partition_prune.out ---

Re: shared-memory based stats collector - v69

2022-04-05 Thread David G. Johnston
On Mon, Apr 4, 2022 at 8:05 PM Andres Freund wrote: > - added an architecture overview comment to the top of pgstat.c - not sure > if > it makes sense to anybody but me (and perhaps Horiguchi-san)? > > I took a look at this, diff attached. Some typos and minor style stuff, plus trying to

Re: shared-memory based stats collector - v66

2022-04-05 Thread Andres Freund
Hi, On 2022-04-02 01:16:48 -0700, Andres Freund wrote: > I just noticed that the code doesn't appear to actually work like that right > now. Whenever the timeout is reached, pgstat_report_stat() is called with > force = true. > > And even if the backend is busy running queries, once there's

Re: should vacuum's first heap pass be read-only?

2022-04-05 Thread Peter Geoghegan
On Tue, Apr 5, 2022 at 1:10 PM Robert Haas wrote: > I had assumed that this would not be the case, because if the page is > being accessed by the workload, it can be pruned - and probably frozen > too, if we wanted to write code for that and spend the cycles on it - > and if it isn't, pruning and

Re: ExecRTCheckPerms() and many prunable partitions

2022-04-05 Thread David Rowley
On Wed, 6 Apr 2022 at 02:27, Greg Stark wrote: > > This is failing regression tests. I don't understand how this patch > could be affecting this test though. Perhaps it's a problem with the > json patches that were committed recently -- but they don't seem to be > causing other patches to fail.

Re: should vacuum's first heap pass be read-only?

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 3:22 PM Peter Geoghegan wrote: > It's not just an enabler of more frequent index vacuuming (for those > indexes that need it the most), though. It's also an enabler of more > frequent lazy_scan_prune processing (in particular setting hint bits > and freezing), which is

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread Nathan Bossart
On Tue, Apr 05, 2022 at 11:25:36AM -0400, Stephen Frost wrote: > Please find attached an updated patch + commit message. Mostly, I just > went through and did a bit more in terms of updating the documentation > and improving the comments (there were some places that were still > worrying about

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-05 Thread Robert Haas
On Fri, Apr 1, 2022 at 11:12 AM Matthias van de Meent wrote: > Here's a new 0001 to keep CFBot happy. This seems like it would conflict with the proposal from http://postgr.es/m/ca+tgmoad8wmn6i1mmuo+4znege3hd57ys8uv8uzm7cneqy3...@mail.gmail.com which I still hope to advance in some form at an

Re: Temporary file access API

2022-04-05 Thread Robert Haas
On Tue, Mar 8, 2022 at 6:12 AM Antonin Houska wrote: > Thanks for your comments, the initial version is attached here. I've been meaning to look at this thread for some time but have not found enough time to do that until just now. And now I have questions... 1. Supposing we accepted this, how

Re: should vacuum's first heap pass be read-only?

2022-04-05 Thread Peter Geoghegan
On Tue, Apr 5, 2022 at 5:44 AM Robert Haas wrote: > The whole idea of decoupling table and index vacuum > supposes that there are situations in which it's worth performing the > first heap pass where we gather the dead line pointers but where it's > not necessary to follow that up as quickly as

Re: Mark all GUC variable as PGDLLIMPORT

2022-04-05 Thread Tom Lane
Robert Haas writes: > Do you care whether your commit > or mine goes in first? I do not. If they're not independent, at least one of us has messed up. I have family commitments on Saturday, so if I don't get mine in on Friday it'll likely not happen before Sunday.

Re: API stability [was: pgsql: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint.]

2022-04-05 Thread Tom Lane
Robert Haas writes: > On Tue, Apr 5, 2022 at 10:32 AM Tom Lane wrote: >> My point is that we want that to happen in HEAD, but it's not okay >> for it to happen in a minor release of a stable branch. > I understand, but I am not sure that I agree. I think that if an > extension stops compiling

Re: Add index scan progress to pg_stat_progress_vacuum

2022-04-05 Thread Imseih (AWS), Sami
>Can't the progress data trivially be inferred by the fact that the worker >completed? Yes, at some point, this idea was experimented with in 0004-Expose-progress-for-the-vacuuming-indexes-cleanup-ph.patch. This patch did the calculation in system_views.sql However, the view is complex

Re: shared-memory based stats collector - v68

2022-04-05 Thread David G. Johnston
On Tuesday, April 5, 2022, Andres Freund wrote: > Hi, > > On 2022-04-05 08:49:36 -0700, David G. Johnston wrote: > > On Mon, Apr 4, 2022 at 7:36 PM Andres Freund wrote: > > > > > > > > I think all this is going to achieve is to making code more > complicated. > > > There > > > is a *single*

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-04-05 Thread Jacob Champion
On Tue, 2022-04-05 at 15:13 +0900, Michael Paquier wrote: > On Wed, Mar 30, 2022 at 04:02:09PM +, Jacob Champion wrote: > > Whether that's a problem in the future entirely depends on whether > > there's some authentication method that considers the empty string a > > sane and meaningful

Re: LogwrtResult contended spinlock

2022-04-05 Thread Alvaro Herrera
Apologies -- I selected the wrong commit to extract the commit message from. Here it is again. I also removed an obsolete /* XXX */ comment. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ >From 0b26f90f1b95f8e9b932eb34bbf9c2a50729cf60 Mon Sep 17 00:00:00 2001

Re: Mark all GUC variable as PGDLLIMPORT

2022-04-05 Thread John Naylor
On Tue, Apr 5, 2022 at 10:06 PM Robert Haas wrote: > > On Tue, Apr 5, 2022 at 10:07 AM Tom Lane wrote: > > Yeah, the frontend error message rework in [1]. That has exactly > > the same constraint that it's likely to break other open patches, > > so it'd be better to do it after the CF cutoff.

Re: Add index scan progress to pg_stat_progress_vacuum

2022-04-05 Thread Andres Freund
On 2022-04-05 16:42:28 +, Imseih (AWS), Sami wrote: > >Why isn't the obvious thing to do here to provide a way to associate > > workers > >with their leaders in shared memory, but to use the existing progress > > fields > >to report progress? Then, when querying progress, the

Re: shared-memory based stats collector - v68

2022-04-05 Thread Andres Freund
Hi, On 2022-04-05 08:49:36 -0700, David G. Johnston wrote: > On Mon, Apr 4, 2022 at 7:36 PM Andres Freund wrote: > > > > > I think all this is going to achieve is to making code more complicated. > > There > > is a *single* non-assert use of accessed_across_databases and now a single > >

Re: Add index scan progress to pg_stat_progress_vacuum

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 12:42 PM Imseih (AWS), Sami wrote: > >Why isn't the obvious thing to do here to provide a way to associate > > workers > >with their leaders in shared memory, but to use the existing progress > > fields > >to report progress? Then, when querying progress, the

Re: wrong fds used for refilenodes after pg_upgrade relfilenode changes Reply-To:

2022-04-05 Thread Robert Haas
On Mon, Apr 4, 2022 at 10:20 PM Thomas Munro wrote: > > The checkpointer never takes heavyweight locks, so the opportunity > > you're describing can't arise. > > Hmm, oh, you probably meant the buffer interlocking > in SyncOneBuffer(). It's true that my most recent patch throws away > more

Re: PostgreSQL 15 Release Management Team (RMT) + Feature Freeze

2022-04-05 Thread Jonathan S. Katz
On 3/26/22 11:10 AM, Jonathan S. Katz wrote: Additionally, the RMT has set the feature freeze date to be April 7, 2022. This is the last day to commit features for PostgreSQL 15. In other words, no new PostgreSQL 15 feature can be committed after April 8 0:00, 2022 AoE[1]. [1]

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-04-05 Thread Peter Geoghegan
On Mon, Apr 4, 2022 at 8:25 PM Peter Geoghegan wrote: > Right. The reason I used WARNINGs was because it matches vaguely > related WARNINGs in vac_update_relstats()'s sibling function, > vacuum_set_xid_limits(). Okay, pushed the relfrozenxid warning patch. Thanks -- Peter Geoghegan

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread Magnus Hagander
On Tue, Apr 5, 2022 at 5:25 PM Stephen Frost wrote: > Greetings, > > * David Steele (da...@pgmasters.net) wrote: > > On 4/4/22 11:42 AM, Nathan Bossart wrote: > > >I noticed a couple of other things that can be removed. Since we no > longer > > >wait on exclusive backup mode during smart

Re: Add index scan progress to pg_stat_progress_vacuum

2022-04-05 Thread Imseih (AWS), Sami
>I nevertheless think that's not acceptable. The whole premise of the > progress >reporting infrastructure is to be low overhead. It's OK to require locking > to >initialize parallel progress reporting, it's definitely not ok to require >locking to report progress. Fair point.

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 10:10 AM Andy Fan wrote: >> > I wanted to have a WAL record spanning multiple WAL files of size, say >> > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table >> > would help here. Please let me know if there's any way to generate >> > such large WAL records.

Re: [PATCH] pg_stat_toast v10

2022-04-05 Thread Robert Haas
On Thu, Mar 31, 2022 at 9:16 AM Gunnar "Nick" Bluth wrote: > That was meant to say "v10", sorry! Hi, >From my point of view, at least, it would be preferable if you'd stop changing the subject line every time you post a new version. Based on the test results in

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread David Steele
On 4/5/22 11:25 AM, Stephen Frost wrote: Please find attached an updated patch + commit message. Mostly, I just went through and did a bit more in terms of updating the documentation and improving the comments (there were some places that were still worrying about the chance of a 'stray'

How to simulate sync/async standbys being closer/farther (network distance) to primary in core postgres?

2022-04-05 Thread Bharath Rupireddy
Hi, I'm thinking if there's a way in core postgres to achieve $subject. In reality, the sync/async standbys can either be closer/farther (which means sync/async standbys can receive WAL at different times) to primary, especially in cloud HA environments with primary in one Availability

Re: Separate the result of \watch for each query execution (psql)

2022-04-05 Thread Robert Haas
On Mon, Mar 21, 2022 at 9:15 PM Andres Freund wrote: > On 2022-02-25 13:23:31 +0900, Noboru Saito wrote: > > I have created a patch that allows you to turn it on and off in \pset. > > The patch unfortunately causes tests to fail: It doesn't seem like the originally proposed design here will be

Re: shared-memory based stats collector - v68

2022-04-05 Thread David G. Johnston
On Mon, Apr 4, 2022 at 7:36 PM Andres Freund wrote: > > I think all this is going to achieve is to making code more complicated. > There > is a *single* non-assert use of accessed_across_databases and now a single > assertion involving it. > > What would having PGSTAT_KIND_CLUSTER and

Re: Printing backtrace of postgres processes

2022-04-05 Thread Robert Haas
On Wed, Mar 30, 2022 at 12:03 PM Justin Pryzby wrote: > On Wed, Mar 30, 2022 at 11:53:52AM -0400, Greg Stark wrote: > > Sadly the cfbot is showing a patch conflict again. It's just a trivial > > conflict in the regression test schedule so I'm not going to update > > the status but it would be

Re: LogwrtResult contended spinlock

2022-04-05 Thread Alvaro Herrera
Here's a v8, where per my previous comment I removed some code that I believe is no longer necessary. I've omitted the patch that renames LogwrtResult subvariables into LogWriteResult/LogWriteFlush; I still think the end result is better after that one, but it's a pretty trivial change that can

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2022-04-05 Thread Yura Sokolov
Good day, David. I'm looking on patch and don't get some moments. `GrantLockLocal` allocates `LOCALLOCKOWNER` and links it into `locallock->locallockowners`. It links it regardless `owner` could be NULL. But then `RemoveLocalLock` does `Assert(locallockowner->owner != NULL);`. Why it should not

Re: Proposal for internal Numeric to Uint64 conversion function.

2022-04-05 Thread Robert Haas
On Thu, Mar 17, 2022 at 4:02 PM Tom Lane wrote: > Pretty much, yeah. I'm way more interested in cleaning up the code > we have than in making things prettier for hypothetical future > call sites. In particular, the problem with writing an API in a > vacuum is that you have little evidence that

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > On 4/4/22 11:42 AM, Nathan Bossart wrote: > >I noticed a couple of other things that can be removed. Since we no longer > >wait on exclusive backup mode during smart shutdown, we can change > >connsAllowed (in postmaster.c) to a boolean

Re: Mark all GUC variable as PGDLLIMPORT

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 10:07 AM Tom Lane wrote: > Yeah, the frontend error message rework in [1]. That has exactly > the same constraint that it's likely to break other open patches, > so it'd be better to do it after the CF cutoff. I think that doing > that concurrently with Robert's thing

Re: [COMMITTERS] pgsql: Allow time delayed standbys and recovery

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 10:58 AM Magnus Hagander wrote: >> Makes sense. I will do this soon if nobody objects. >> >> I'm mildly uncomfortable with the phrase "WAL records generated over >> the delay period" because it seems a bit imprecise, but I'm not sure >> what would be better and I think the

Re: [COMMITTERS] pgsql: Allow time delayed standbys and recovery

2022-04-05 Thread Magnus Hagander
On Tue, Apr 5, 2022 at 4:54 PM Robert Haas wrote: > On Tue, Apr 5, 2022 at 8:41 AM Thom Brown wrote: > > I know it's been 8 years, but I still think it would be a useful note > > to add to the docs. > Many points for bringing that one back :) Makes sense. I will do this soon if nobody

Re: [COMMITTERS] pgsql: Allow time delayed standbys and recovery

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 8:41 AM Thom Brown wrote: > I know it's been 8 years, but I still think it would be a useful note > to add to the docs. Makes sense. I will do this soon if nobody objects. I'm mildly uncomfortable with the phrase "WAL records generated over the delay period" because it

Re: API stability [was: pgsql: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint.]

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 10:32 AM Tom Lane wrote: > Robert Haas writes: > > On Tue, Apr 5, 2022 at 10:17 AM Tom Lane wrote: > >> Renaming it would constitute an API break, which is if anything worse > >> than an ABI break. > > > I don't think so, because an API break will cause a compilation > >

Re: API stability [was: pgsql: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint.]

2022-04-05 Thread Tom Lane
Robert Haas writes: > On Tue, Apr 5, 2022 at 10:17 AM Tom Lane wrote: >> Renaming it would constitute an API break, which is if anything worse >> than an ABI break. > I don't think so, because an API break will cause a compilation > failure, which an extension author can easily fix. My point

Re: API stability [was: pgsql: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint.]

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 10:17 AM Tom Lane wrote: > Renaming it would constitute an API break, which is if anything worse > than an ABI break. I don't think so, because an API break will cause a compilation failure, which an extension author can easily fix. > While we're complaining at you, let

Re: ExecRTCheckPerms() and many prunable partitions

2022-04-05 Thread Greg Stark
This is failing regression tests. I don't understand how this patch could be affecting this test though. Perhaps it's a problem with the json patches that were committed recently -- but they don't seem to be causing other patches to fail. diff -U3

Re: API stability [was: pgsql: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint.]

2022-04-05 Thread Tom Lane
Robert Haas writes: > On Tue, Apr 5, 2022 at 9:02 AM Markus Wanner > wrote: >> And for this specific case: Is it worth reverting this change and >> applying a fully backwards compatible fix, instead? > I think it's normally our policy to avoid changing definitions of > accessible structs in

Re: Mingw task for Cirrus CI

2022-04-05 Thread Andrew Dunstan
On 4/4/22 16:41, Andres Freund wrote: > Hi, > > On 2022-03-30 17:26:18 -0700, Andres Freund wrote: >> Hi, >> >> On 2022-03-22 19:00:42 +0300, Melih Mutlu wrote: >>> Rebased it. >>> I also removed the temp installation task and >>> used NoDefaultCurrentDirectoryInExePath env variable instead. >>

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Andy Fan
Hi, On Tue, Apr 5, 2022 at 9:46 PM Alvaro Herrera wrote: > On 2022-Apr-05, Bharath Rupireddy wrote: > > > Hi, > > > > I wanted to have a WAL record spanning multiple WAL files of size, say > > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table > > would help here. Please let me

Re: Mark all GUC variable as PGDLLIMPORT

2022-04-05 Thread Tom Lane
Andrew Dunstan writes: > On 3/30/22 14:37, Robert Haas wrote: >> @RMT: Andres proposed upthread that we should plan to do this just >> after feature freeze. Accordingly I propose to commit at least 0002 >> and perhaps 0001 if people want it just after feature freeze. I >> therefore ask that the

Re: API stability [was: pgsql: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint.]

2022-04-05 Thread Robert Haas
On Tue, Apr 5, 2022 at 9:02 AM Markus Wanner wrote: > And for this specific case: Is it worth reverting this change and > applying a fully backwards compatible fix, instead? I think it's normally our policy to avoid changing definitions of accessible structs in back branches, except that we

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Alvaro Herrera
On 2022-Apr-05, Bharath Rupireddy wrote: > Hi, > > I wanted to have a WAL record spanning multiple WAL files of size, say > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table > would help here. Please let me know if there's any way to generate > such large WAL records. It's

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Matthias van de Meent
On Tue, 5 Apr 2022 at 15:13, Bharath Rupireddy wrote: > > Hi, > > I wanted to have a WAL record spanning multiple WAL files of size, say > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table > would help here. Please let me know if there's any way to generate > such large WAL

Re: Handle infinite recursion in logical replication setup

2022-04-05 Thread Ashutosh Bapat
On Tue, Apr 5, 2022 at 6:21 AM Peter Smith wrote: > > Here are my comments for the latest patch v6-0001. > > (I will post my v6-0002 review comments separately) > > PATCH v6-0001 comments > == > > 1.1 General - Option name > > I still feel like the option name is not ideal.

Re: Mark all GUC variable as PGDLLIMPORT

2022-04-05 Thread Andrew Dunstan
On 3/30/22 14:37, Robert Haas wrote: > On Fri, Feb 18, 2022 at 7:02 PM Andres Freund wrote: >> On 2022-02-15 08:06:58 -0800, Andres Freund wrote: >>> The more I think about it the more I'm convinced that if we want to do this, >>> we should do it for variables and functions. >> Btw, if we were

How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Bharath Rupireddy
Hi, I wanted to have a WAL record spanning multiple WAL files of size, say 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table would help here. Please let me know if there's any way to generate such large WAL records. Thoughts? Regards, Bharath Rupireddy.

A fastpath for TransactionIdIsCurrentTransactionId

2022-04-05 Thread Andy Fan
Hi: In my recent work, I want to check if the xmin for all the tuples is CurrentTransactioniId, then I found we can improve the fastpath for TransactionIdIsCurrentTransactionId like below, would it be safe? This would be helpful if we have lots of sub transactionId. diff --git

API stability [was: pgsql: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint.]

2022-04-05 Thread Markus Wanner
On 24.03.22 20:32, Robert Haas wrote: Fix possible recovery trouble if TRUNCATE overlaps a checkpoint. This patch changed the delayChkpt field of struct PGPROC from bool to int. Back-porting this change could be considered an API breaking change for extensions using this field. I'm not

Re: Window Function "Run Conditions"

2022-04-05 Thread Andy Fan
On Tue, Apr 5, 2022 at 7:49 PM David Rowley wrote: > On Tue, 5 Apr 2022 at 19:38, Andy Fan wrote: > > 1. We can do more on PASSTHROUGH, we just bypass the window function > > currently, but IIUC we can ignore all of the following tuples in > current partition > > once we go into this mode.

  1   2   >