Re: Table AM Interface Enhancements

2024-04-10 Thread Andres Freund
Hi, On 2024-04-10 16:50:44 -0400, Melanie Plageman wrote: > This brings up a question about the prefetching. We never had to have > this discussion for sequential scan streaming read because it didn't > (and still doesn't) do prefetching. But, if we push the streaming read > code down into the hea

allow changing autovacuum_max_workers without restarting

2024-04-10 Thread Nathan Bossart
I frequently hear about scenarios where users with thousands upon thousands of tables realize that autovacuum is struggling to keep up. When they inevitably go to bump up autovacuum_max_workers, they discover that it requires a server restart (i.e., downtime) to take effect, causing further frustr

Re: Fix possible dereference null pointer (src/backend/replication/logical/reorderbuffer.c)

2024-04-10 Thread Heikki Linnakangas
On 10/04/2024 21:07, Ranier Vilela wrote: Hi, Per Coverity. The function ReorderBufferTXNByXid, can return NULL when the parameter *create* is false. In the functions ReorderBufferSetBaseSnapshot and ReorderBufferXidHasBaseSnapshot, the second call to ReorderBufferTXNByXid, pass false to *crea

Re: psql: Greatly speed up "\d tablename" when not using regexes

2024-04-10 Thread Jelte Fennema-Nio
On Wed, 10 Apr 2024 at 22:11, Tom Lane wrote: > There may be an argument for psql to do what you suggest, > but so far it seems like duplicative complication. > > If there's a case you can demonstrate where "\d foo" doesn't optimize > into an indexscan, we should look into exactly why that's happe

Re: psql: Greatly speed up "\d tablename" when not using regexes

2024-04-10 Thread Tom Lane
Jelte Fennema-Nio writes: > On Wed, 10 Apr 2024 at 22:11, Tom Lane wrote: >> If there's a case you can demonstrate where "\d foo" doesn't optimize >> into an indexscan, we should look into exactly why that's happening, >> because I think the cause must be more subtle than this. > Hmm, okay so I

Re: recovery modules

2024-04-10 Thread Nathan Bossart
rebased -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 5897631d5f09032565d92d5b8547baf3d24eef87 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v21 1/5] introduce routine for checking mutually exclusive string GUCs --- s

Re: Allow non-superuser to cancel superuser tasks.

2024-04-10 Thread Michael Paquier
On Wed, Apr 10, 2024 at 10:00:34AM -0500, Nathan Bossart wrote: > Isn't it relatively easy to discover this same information today via > pg_stat_progress_vacuum? That has the following code: > > /* Value available to all callers */ > values[0] = Int32GetDatum(beentry->

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Michael Paquier
On Thu, Apr 11, 2024 at 12:20:55AM +0300, Heikki Linnakangas wrote: > To move this forward, here's a patch to switch to a pairing heap. In my very > quick testing, with the performance test cases posted earlier in this thread > [1] [2], I'm seeing no meaningful performance difference between this a

Re: Cutting support for OpenSSL 1.0.1 and 1.0.2 in 17~?

2024-04-10 Thread Michael Paquier
On Wed, Apr 10, 2024 at 09:31:16AM +0200, Peter Eisentraut wrote: > I think it might be better to separate this into two steps: > > 1. Move to 1.1.0. This is an API update. Change OPENSSL_API_COMPAT, and > remove a bunch of code that no longer needs to be conditional. We could > check for a rep

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-04-10 Thread Heikki Linnakangas
On 07/04/2024 00:52, Alexander Korotkov wrote: On Fri, Apr 5, 2024 at 9:15 PM Alvaro Herrera wrote: I'm still concerned that WaitLSNCleanup is only called in ProcKill. Does this mean that if a process throws an error while waiting, it'll not get cleaned up until it exits? Maybe this is not a b

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Heikki Linnakangas
On 11/04/2024 01:37, Michael Paquier wrote: On Thu, Apr 11, 2024 at 12:20:55AM +0300, Heikki Linnakangas wrote: To move this forward, here's a patch to switch to a pairing heap. In my very quick testing, with the performance test cases posted earlier in this thread [1] [2], I'm seeing no meaning

Re: post-freeze damage control

2024-04-10 Thread David Steele
On 4/10/24 09:50, Michael Paquier wrote: On Wed, Apr 10, 2024 at 09:29:38AM +1000, David Steele wrote: Even so, only keeping WAL for the last backup is a dangerous move in any case. Lots of things can happen to a backup (other than bugs in the software) so keeping WAL back to the last full (o

Re: Add notes to pg_combinebackup docs

2024-04-10 Thread David Steele
On 4/9/24 19:44, Tomas Vondra wrote: On 4/9/24 09:59, Martín Marqués wrote: Hello, While doing some work/research on the new incremental backup feature some limitations were not listed in the docs. Mainly the fact that pg_combienbackup works with plain format and not tar. Right. The docs mo

Re: post-freeze damage control

2024-04-10 Thread Tom Kincaid
> > > Yeah, that's an excellent practive, but is why I'm less worried for > > this feature. The docs at [1] caution about "not to remove earlier > > backups if they might be needed when restoring later incremental > > backups". Like Alvaro said, should we insist a bit more about the WAL > > reten

Re: Is this a problem in GenericXLogFinish()?

2024-04-10 Thread Michael Paquier
On Wed, Apr 10, 2024 at 03:28:22PM +0530, Amit Kapila wrote: > I can understand this comment as I am aware of this code but not sure > it would be equally easy for the people first time looking at this > code. One may try to find the equivalent assertion in > _hash_freeovflpage(). The alternative c

Re: Issue with the PRNG used by Postgres

2024-04-10 Thread Robert Haas
On Wed, Apr 10, 2024 at 4:40 PM Tom Lane wrote: > I'm not worried about it being slower, but about whether it could > report "stuck spinlock" in cases where the existing code succeeds. > While that seems at least theoretically possible, it seems like > if you hit it you have got problems that need

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2024-04-10 Thread Michael Paquier
On Tue, Apr 09, 2024 at 09:33:49AM +0300, Andrey M. Borodin wrote: > As far as I understand CF entry [0] is committed? I understand that > there are some open followups, but I just want to determine correct > CF item status... So much work has happened on this thread with things that has been com

Re: CI and test improvements

2024-04-10 Thread Michael Paquier
On Mon, Apr 08, 2024 at 05:54:10PM +0300, Andrey M. Borodin wrote: > Justin, Peter, I can't determine actual status of the CF entry > [0]. May I ask someone of you to move patch to next CF or close as > committed? 0002 is the only thing committed as of 21a71648d39f. I can see the value in 0001, b

Re: Issue with the PRNG used by Postgres

2024-04-10 Thread Tom Lane
Robert Haas writes: > I just want to mention that I have heard of "stuck spinlock" happening > in production just because the server was busy. And I think that's not > intended. The timeout is supposed to be high enough that you only hit > it if there's a bug in the code. At least AIUI. But it isn

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Masahiko Sawada
Hi, Sorry for the late reply, I took two days off. On Thu, Apr 11, 2024 at 6:20 AM Heikki Linnakangas wrote: > > On 10/04/2024 08:31, Amit Kapila wrote: > > On Wed, Apr 10, 2024 at 11:00 AM Heikki Linnakangas wrote: > >> > >> On 10/04/2024 07:45, Michael Paquier wrote: > >>> On Tue, Apr 09, 202

Re: SET ROLE documentation improvement

2024-04-10 Thread Michael Paquier
On Tue, Apr 09, 2024 at 09:21:39AM +0300, Andrey M. Borodin wrote: > Can I ask you please to help me with determining status of CF item > [0]. Is it committed or there's something to move to next CF? Only half of the patch has been applied as of 3330a8d1b792. Yurii and Nathan, could you follow up

pg_combinebackup does not detect missing files

2024-04-10 Thread David Steele
Hackers, I've been playing around with the incremental backup feature trying to get a sense of how it can be practically used. One of the first things I always try is to delete random files and see what happens. You can delete pretty much anything you want from the most recent incremental ba

RE: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Hayato Kuroda (Fujitsu)
Dear Heikki, I also prototyped the idea, which has almost the same shape. I attached just in case, but we may not have to see. Few comments based on the experiment. ``` + /* txn_heap is ordered by transaction size */ + buffer->txn_heap = pairingheap_allocate(ReorderBufferTXNSizeCompa

Re: post-freeze damage control

2024-04-10 Thread David Steele
On 4/11/24 10:23, Tom Kincaid wrote: The extensive Beta process we have can be used to build confidence we need in a feature that has extensive review and currently has no known issues or outstanding objections. I did have objections, here [1] and here [2]. I think the complexity, space req

Re: Issue with the PRNG used by Postgres

2024-04-10 Thread Tom Lane
I wrote: > I'm not worried about it being slower, but about whether it could > report "stuck spinlock" in cases where the existing code succeeds. On fourth thought ... the number of tries to acquire the lock, or in this case number of tries to observe the lock free, is not NUM_DELAYS but NUM_DELAY

Re: Potential stack overflow in incremental base backup

2024-04-10 Thread Thomas Munro
On Thu, Apr 11, 2024 at 12:11 AM Robert Haas wrote: > On Wed, Apr 10, 2024 at 6:21 AM Thomas Munro wrote: > > Could we just write the blocks directly into the output array, and > > then transpose them directly in place if start_blkno > 0? See > > attached. I may be missing something, but the on

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-04-10 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > One naive idea is that on the publisher we can remember whether the > prepare has been sent and if so then only send commit_prepared, > otherwise send the entire transaction. On the subscriber-side, we > somehow, need to ensure before applying the first change whether the > correspond

Re: Requiring LLVM 14+ in PostgreSQL 18

2024-04-10 Thread Thomas Munro
On Wed, Apr 10, 2024 at 1:38 PM Thomas Munro wrote: > Therefore, some time after the tree re-opens for hacking, we could rip > out a bunch of support code for LLVM 10-13, and then rip out support > for pre-opaque-pointer mode. Please see attached. ... or of course closer to the end of the cycle

Re: Can't find not null constraint, but \d+ shows that

2024-04-10 Thread Tender Wang
Alvaro Herrera 于2024年4月10日周三 21:58写道: > It turns out that trying to close all holes that lead to columns marked > not-null without a pg_constraint row is not possible within the ALTER > TABLE framework, because it can happen outside it also. Consider this > > CREATE DOMAIN dom1 AS integer; > CRE

Re: Incorrect handling of IS [NOT] NULL quals on inheritance parents

2024-04-10 Thread David Rowley
On Wed, 10 Apr 2024 at 19:12, Richard Guo wrote: > And I think recording NOT NULL columns for traditional inheritance > parents can be error-prone for some future optimization where we look > at an inheritance parent's notnullattnums and make decisions based on > the assumption that the included c

Re: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Masahiko Sawada
On Thu, Apr 11, 2024 at 10:32 AM Masahiko Sawada wrote: > > Hi, > > Sorry for the late reply, I took two days off. > > On Thu, Apr 11, 2024 at 6:20 AM Heikki Linnakangas wrote: > > > > On 10/04/2024 08:31, Amit Kapila wrote: > > > On Wed, Apr 10, 2024 at 11:00 AM Heikki Linnakangas > > > wrote:

Re: sql/json remaining issue

2024-04-10 Thread jian he
On Wed, Apr 10, 2024 at 4:39 PM Amit Langote wrote: > > > Attached is a bit more polished version of that, which also addresses > the error messages in JsonPathQuery() and JsonPathValue(). I noticed > that there was comment I had written at one point during JSON_TABLE() > hacking that said that w

Re: DROP DATABASE is interruptible

2024-04-10 Thread Thomas Munro
On Tue, Mar 12, 2024 at 9:00 PM Alexander Lakhin wrote: > I see two backends waiting: > law 2420132 2420108 0 09:05 ?00:00:00 postgres: node: law > postgres [local] DROP DATABASE waiting > law 2420135 2420108 0 09:05 ?00:00:00 postgres: node: law > postgres [local] st

Re: Synchronizing slots from primary to standby

2024-04-10 Thread Amit Kapila
On Wed, Apr 10, 2024 at 5:28 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, April 4, 2024 5:37 PM Amit Kapila > wrote: > > > > BTW, while thinking on this one, I > > noticed that in the function LogicalConfirmReceivedLocation(), we first > > update > > the disk copy, see comment [1] and then i

Re: Issue with the PRNG used by Postgres

2024-04-10 Thread Andrey M. Borodin
> On 10 Apr 2024, at 21:48, Parag Paul wrote: > > Yes, the probability of this happening is astronomical, but in production > with 128 core servers with 7000 max_connections, with petabyte scale data, > this did repro 2 times in the last month. We had to move to a local approach > to manage

apply_scanjoin_target_to_paths and partitionwise join

2024-04-10 Thread Ashutosh Bapat
Hi All, Per below code and comment in apply_scanjoin_target_to_paths(), the function zaps all the paths of a partitioned relation. /* * If the rel is partitioned, we want to drop its existing paths and * generate new ones. This function would still be correct if we kept the * existing paths: we'd

Re: Can't find not null constraint, but \d+ shows that

2024-04-10 Thread jian he
On Wed, Apr 10, 2024 at 2:10 PM jian he wrote: > > DROP TABLE if exists notnull_tbl2; > CREATE TABLE notnull_tbl2 (c0 int generated by default as IDENTITY, c1 int); > ALTER TABLE notnull_tbl2 ADD CONSTRAINT Q PRIMARY KEY(c0, c1); > ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c0_not_null;

Re: Incorrect handling of IS [NOT] NULL quals on inheritance parents

2024-04-10 Thread Richard Guo
On Thu, Apr 11, 2024 at 10:23 AM David Rowley wrote: > On Wed, 10 Apr 2024 at 19:12, Richard Guo wrote: > > And I think recording NOT NULL columns for traditional inheritance > > parents can be error-prone for some future optimization where we look > > at an inheritance parent's notnullattnums a

Re: apply_scanjoin_target_to_paths and partitionwise join

2024-04-10 Thread Ashutosh Bapat
On Thu, Apr 11, 2024 at 12:07 PM Ashutosh Bapat < ashutosh.bapat@gmail.com> wrote: > Hi All, > Per below code and comment in apply_scanjoin_target_to_paths(), the > function zaps all the paths of a partitioned relation. > /* > * If the rel is partitioned, we want to drop its existing paths and

<    1   2