Re: automating RangeTblEntry node support

2024-02-19 Thread Peter Eisentraut
On 18.02.24 00:06, Matthias van de Meent wrote: I'm not sure that the cleanup which is done when changing a RTE's rtekind is also complete enough for this purpose. Things like inline_cte_walker change the node->rtekind, which could leave residual junk data in fields that are currently dropped

pg_restore option --clean

2024-02-19 Thread Fabrice Chapuis
Hi, The --clean option of pg_restore allows you to replace an object before being imported. However, dependencies such as foreign keys or views prevent the deletion of the object. Is there a way to add the cascade option to force the deletion? Thanks for helping Fabrice

Re: Switching XLog source from archive to streaming when primary available

2024-02-19 Thread Bharath Rupireddy
On Tue, Feb 20, 2024 at 11:54 AM Japin Li wrote: > > > On Tue, 20 Feb 2024 at 13:40, Bharath Rupireddy > wrote: > > On Mon, Feb 19, 2024 at 8:25 PM Japin Li wrote: > >> [2] > >> +# Ensure checkpoint doesn't come in our way > >> +$primary->append_conf('postgresql.conf', qq( > >> +

Re: confirmed flush lsn seems to be move backward in certain error cases

2024-02-19 Thread vignesh C
On Sat, 17 Feb 2024 at 12:03, Amit Kapila wrote: > > On Fri, Feb 16, 2024 at 5:53 PM vignesh C wrote: > > > > > > After the insert operation is replicated to the subscriber, the > > subscriber will set the lsn value sent by the publisher in the > > replication origin (in my case it was

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-19 Thread Masahiko Sawada
On Mon, Feb 19, 2024 at 7:47 PM John Naylor wrote: > > On Mon, Feb 19, 2024 at 9:02 AM Masahiko Sawada wrote: > > > > I think that vacuum and tidbitmap (and future users) would end up > > having the same max block size calculation. And it seems slightly odd > > layering to me that

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-02-19 Thread Bharath Rupireddy
On Fri, Feb 9, 2024 at 1:12 PM Bertrand Drouvot wrote: > > I think "conflict" is an important topic and does contain several reasons. The > slot "first" conflict and then leads to slot "invalidation". > > > They both are the same internally, so why > > confuse the users? > > I don't think that

Re: Patch: Add parse_type Function

2024-02-19 Thread jian he
On Tue, Feb 20, 2024 at 11:06 AM David E. Wheeler wrote: > > LOL, I missed that, just wrote it myself in the last hour. :-) v6 attached. > +SELECT to_regtypemod('interval nonesuch'); -- grammar error expected +ERROR: syntax error at or near "nonesuch" +LINE 1: SELECT to_regtypemod('interval

Re: Switching XLog source from archive to streaming when primary available

2024-02-19 Thread Japin Li
On Tue, 20 Feb 2024 at 13:40, Bharath Rupireddy wrote: > On Mon, Feb 19, 2024 at 8:25 PM Japin Li wrote: >> [2] >> +# Ensure checkpoint doesn't come in our way >> +$primary->append_conf('postgresql.conf', qq( >> +min_wal_size = 2MB >> +max_wal_size = 1GB >> +checkpoint_timeout = 1h

Re: Shared detoast Datum proposal

2024-02-19 Thread Andy Fan
Hi, I didn't another round of self-review. Comments, variable names, the order of function definition are improved so that it can be read as smooth as possible. so v6 attached. -- Best Regards Andy Fan >From f2e7772228e8a18027b9c29f10caba9c6570d934 Mon Sep 17 00:00:00 2001 From: "yizhi.fzh"

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2024-02-19 Thread Bharath Rupireddy
On Sat, Feb 17, 2024 at 10:27 AM Bharath Rupireddy wrote: > > On Fri, Feb 16, 2024 at 11:01 PM Jeff Davis wrote: > > > > > Here, I'm with v23 patch set: > > > > Thank you, I'll look at these. > > Thanks. Here's the v24 patch set after rebasing. Ran pgperltidy on the new TAP test file added.

Re: Add test module for verifying backtrace functionality

2024-02-19 Thread Bharath Rupireddy
On Tue, Feb 13, 2024 at 2:11 AM Bharath Rupireddy wrote: > > Hi, > > Postgres has a good amount of code for dealing with backtraces - two > GUCs backtrace_functions and backtrace_on_internal_error, > errbacktrace; all of which use core function set_backtrace from > elog.c. I've not seen this code

Re: partitioning and identity column

2024-02-19 Thread Alexander Lakhin
20.02.2024 07:57, Ashutosh Bapat wrote: Could you please name functions, which you suspect, for me to recheck them? Perhaps we should consider fixing all of such functions, in light of b0f7dd915 and d57b7cc33... Looks like the second commit has fixed all other places I knew except Identity

Re: logical decoding and replication of sequences, take 2

2024-02-19 Thread Amit Kapila
On Thu, Dec 21, 2023 at 6:47 PM Tomas Vondra wrote: > > On 12/19/23 13:54, Christophe Pettus wrote: > > Hi, > > > > I wanted to hop in here on one particular issue: > > > >> On Dec 12, 2023, at 02:01, Tomas Vondra > >> wrote: > >> - desirability of the feature: Random IDs (UUIDs etc.) are

Re: Switching XLog source from archive to streaming when primary available

2024-02-19 Thread Bharath Rupireddy
On Mon, Feb 19, 2024 at 8:25 PM Japin Li wrote: > > > Strengthened tests a bit by using recovery_min_apply_delay to mimic > > standby spending some time fetching from archive. PSA v18 patch. > > Here are some minor comments: Thanks for taking a look at it. > [1] > +primary). However,

Re: JIT compilation per plan node

2024-02-19 Thread David Rowley
On Tue, 20 Feb 2024 at 18:31, Tom Lane wrote: > FWIW, I seriously doubt that an extra walk of the plan tree is even > measurable compared to the number of cycles JIT compilation will > expend if it's called. So I don't buy your argument here. > We would be better off to do this in a way that's

Re: Fix race condition in InvalidatePossiblyObsoleteSlot()

2024-02-19 Thread Michael Paquier
On Tue, Feb 20, 2024 at 08:51:17AM +0900, Michael Paquier wrote: > Prefixing these with "initial_" is fine, IMO. That shows the > intention that these come from the slot's data before doing the > termination. So I'm OK with what's been proposed in v3. I was looking at that a second time, and

Re: JIT compilation per plan node

2024-02-19 Thread Tom Lane
David Rowley writes: > On Tue, 20 Feb 2024 at 05:26, Tomas Vondra > wrote: >> Wouldn't it be simpler to just build the plan as we do now, and then >> have an expression_tree_walker that walks the complete plan top-down, >> inspects the nodes, enables JIT where appropriate and so on? That can >>

Re: JIT compilation per plan node

2024-02-19 Thread David Rowley
On Tue, 20 Feb 2024 at 05:26, Tomas Vondra wrote: > I doubt CreatePlanContext is a great way to achieve this. For one, it > breaks the long-standing custom that PlannerInfo is the first parameter, > usually followed by RelOptInfo, etc. CreatePlanContext is added to some > functions (but not all),

Re: Memory consumed by paths during partitionwise join planning

2024-02-19 Thread Ashutosh Bapat
On Tue, Feb 20, 2024 at 8:19 AM Andrei Lepikhov wrote: > > On 19/2/2024 19:25, Ashutosh Bapat wrote: > > On Fri, Feb 16, 2024 at 8:42 AM Andrei Lepikhov > > wrote: > >> Live example: right now, I am working on the code like MSSQL has - a > >> combination of NestLoop and HashJoin paths and

Re: logical decoding and replication of sequences, take 2

2024-02-19 Thread Robert Haas
On Fri, Feb 16, 2024 at 1:57 AM Tomas Vondra wrote: > For me, the part that I feel most uneasy about is the decoding while the > snapshot is still being built (and can flip to consistent snapshot > between the relfilenode creation and sequence change, confusing the > logic that decides which

Re: partitioning and identity column

2024-02-19 Thread Ashutosh Bapat
On Mon, Feb 19, 2024 at 8:30 PM Alexander Lakhin wrote: > > Hello Ashutosh, > > 19.02.2024 15:17, Ashutosh Bapat wrote: > > > >> Functions ATExecAddIdentity() and ATExecDropIdentity() are recursive too, > >> so I think they can be exploited as well. > > not just Identity related functions, but

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-02-19 Thread Amit Kapila
On Tue, Feb 20, 2024 at 5:04 AM Ian Lawrence Barwick wrote: > > > With the addition of "pg_sync_replication_slots()", there is now a use-case > for > including "dbname" in "primary_conninfo" and the docs have changed from > stating [1]: > > Do not specify a database name in the

Re: RFC: Logging plan of the running query

2024-02-19 Thread Robert Haas
On Fri, Feb 16, 2024 at 12:29 AM Andres Freund wrote: > If we went with something like tht approach, I think we'd have to do something > like redirecting node->ExecProcNode to a wrapper, presumably from within a > CFI. That wrapper could then implement the explain support, without slowing > down

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
On 20/2/2024 04:51, Tom Lane wrote: Tomas Vondra writes: On 2/19/24 16:45, Tom Lane wrote: Tomas Vondra writes: For example, I don't think we expect selectivity functions to allocate long-lived objects, right? So maybe we could run them in a dedicated memory context, and reset it

Re: WIP Incremental JSON Parser

2024-02-19 Thread Andrew Dunstan
On 2024-01-26 Fr 12:15, Andrew Dunstan wrote: On 2024-01-24 We 13:08, Robert Haas wrote: Maybe you should adjust your patch to dump the manifests into the log file with note(). Then when cfbot runs on it you can see exactly what the raw file looks like. Although I wonder if it's possible

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
On 20/2/2024 11:03, jian he wrote: Neither the code comments nor the commit message really explain the design idea here. That's unfortunate, principally because it makes review difficult. I'm very skeptical about the idea of using JumbleExpr for any part of this. It seems fairly expensive, and

Re: speed up a logical replica setup

2024-02-19 Thread Shlok Kyal
Hi, On Tue, 20 Feb 2024 at 06:59, Euler Taveira wrote: > > On Mon, Feb 19, 2024, at 7:22 AM, Shlok Kyal wrote: > > I have reviewed the v21 patch. And found an issue. > > Initially I started the standby server with a new postgresql.conf file > (not the default postgresql.conf that is present in

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
On 19/2/2024 20:47, Tomas Vondra wrote: On 9/8/23 07:11, Lepikhov Andrei wrote: Just for comparison, without partitioning: elems 1 1E1 1E2 1E3 1E4 master: 12kB14kB37kB266kB 2.5MB patched:12kB11.5kB 13kB24kB

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread jian he
On Mon, Feb 19, 2024 at 4:35 PM Andrei Lepikhov wrote: > > In attachment - v17 for both patches. As I see it, the only general > explanation of the idea is not addressed. I'm not sure how deeply we > should explain it. > On Tue, Nov 28, 2023 at 5:04 AM Robert Haas wrote: > > On Mon, Nov 27,

Re: Synchronizing slots from primary to standby

2024-02-19 Thread Amit Kapila
On Tue, Feb 20, 2024 at 8:25 AM Masahiko Sawada wrote: > > Some comments not related to the patch but to the existing code: > > --- > It might have already been discussed but is the > src/backend/replication/logical the right place for the slocsync.c? If > it's independent of logical

Re: Synchronizing slots from primary to standby

2024-02-19 Thread shveta malik
On Tue, Feb 20, 2024 at 8:25 AM Masahiko Sawada wrote: > > > I've reviewed the v91 patch. Here are random comments: Thanks for the comments. > --- > /* > * Checks the remote server info. > * > - * We ensure that the 'primary_slot_name' exists on the remote server and the > - * remote

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
On 19/2/2024 19:53, Ranier Vilela wrote: v17-0002 1) move the vars *arrayconst and *dest, to after if, to avoid makeNode (palloc). + Const   *arrayconst; + ScalarArrayOpExpr  *dest; + + pd = (PredicatesData *) lfirst(lc); + if (pd->elems == NIL) + /* The index doesn't participate in this

Re: Patch: Add parse_type Function

2024-02-19 Thread David E. Wheeler
On Feb 19, 2024, at 21:58, Erik Wienhold wrote: > See the patch I wrote for my benchmarks. But it's pretty easy anyway to > cut down parse_type() ;) LOL, I missed that, just wrote it myself in the last hour. :-) v6 attached. > But you don't actually need reformat_type() in pgTAP. You can

Re: Patch: Add parse_type Function

2024-02-19 Thread Erik Wienhold
On 2024-02-19 23:59 +0100, David E. Wheeler wrote: > On Feb 19, 2024, at 15:47, Tom Lane wrote: > > >> 1. Add a to_regtypmod() for those who just want the typemod. > > > > Seems like there's a good case for doing that. > > I’ll work on that. See the patch I wrote for my benchmarks. But it's

Re: Synchronizing slots from primary to standby

2024-02-19 Thread Masahiko Sawada
On Mon, Feb 19, 2024 at 9:59 PM shveta malik wrote: > > On Mon, Feb 19, 2024 at 5:32 PM Amit Kapila wrote: > > > > Few comments on 0001 > > Thanks for the feedback. > > > > > 1. I think it is better to error out when the valid GUC or option is > > not set in

Re: Memory consumed by paths during partitionwise join planning

2024-02-19 Thread Andrei Lepikhov
On 19/2/2024 19:25, Ashutosh Bapat wrote: On Fri, Feb 16, 2024 at 8:42 AM Andrei Lepikhov wrote: Live example: right now, I am working on the code like MSSQL has - a combination of NestLoop and HashJoin paths and switching between them in real-time. It requires both paths in the path list at

Support boolcol IS [NOT] UNKNOWN in partition pruning

2024-02-19 Thread David Rowley
While working on 4c2369ac5, I noticed there's close to as much code to disallow BooleanTests in the form of "IS UNKNOWN" and "IS NOT UNKNOWN" in partition pruning as it would take to allow pruning to work for these. The attached makes it work. David From b9f3ff909652c96f1f0dced9e1165ffa8c93c7f1

Re: Thoughts about NUM_BUFFER_PARTITIONS

2024-02-19 Thread wenhui qiu
Hi Heikki Linnakangas I saw git log found this commit: https://github.com/postgres/postgres/commit/3acc10c997f916f6a741d0b4876126b7b08e3892 ,I don't seem to see an email discussing this commit. As the commit log tells us, we don't know exactly how large a value is optimal, and I believe it's

Re: Thoughts about NUM_BUFFER_PARTITIONS

2024-02-19 Thread wenhui qiu
Hi Japlin Li Thank you for such important information ! Got it Japin Li 于2024年2月19日周一 10:26写道: > > On Mon, 19 Feb 2024 at 00:56, Tomas Vondra > wrote: > > On 2/18/24 03:30, Li Japin wrote: > >> > >> I find it seems need to change MAX_SIMUL_LWLOCKS if we enlarge the > NUM_BUFFER_PARTITIONS,

Re: speed up a logical replica setup

2024-02-19 Thread Euler Taveira
On Mon, Feb 19, 2024, at 7:22 AM, Shlok Kyal wrote: > I have reviewed the v21 patch. And found an issue. > > Initially I started the standby server with a new postgresql.conf file > (not the default postgresql.conf that is present in the instance). > pg_ctl -D ../standby start -o "-c

Re: serial not accepted as datatype in ALTER TABLE ... ALTER COLUMN

2024-02-19 Thread Andy Fan
Ashutosh Bapat writes: > On Sun, Feb 18, 2024 at 1:59 PM Andy Fan wrote: >> >> >> I tried your idea with the attatchment, it is still in a drafted state >> but it can be used as a prove-of-concept and for better following >> communicating. Just one point needs to metion is serial implies >>

Re: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-02-19 Thread Jelte Fennema-Nio
On Tue, 20 Feb 2024 at 00:34, Ian Lawrence Barwick wrote: > With the addition of "pg_sync_replication_slots()", there is now a use-case > for > including "dbname" in "primary_conninfo" and the docs have changed from > stating [1]: > > Do not specify a database name in the primary_conninfo

Re: Fix race condition in InvalidatePossiblyObsoleteSlot()

2024-02-19 Thread Michael Paquier
On Mon, Feb 19, 2024 at 09:49:24AM +, Bertrand Drouvot wrote: > On Mon, Feb 19, 2024 at 01:45:16PM +0530, Bharath Rupireddy wrote: >> Prefix 'initial_' makes the variable names a bit longer, I think we >> can just use effective_xmin, catalog_effective_xmin and restart_lsn, >> the code updating

Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-02-19 Thread Ian Lawrence Barwick
Hi Hi With the addition of "pg_sync_replication_slots()", there is now a use-case for including "dbname" in "primary_conninfo" and the docs have changed from stating [1]: Do not specify a database name in the primary_conninfo string. to [2]: For replication slot synchronization (see

Re: speed up a logical replica setup

2024-02-19 Thread Euler Taveira
On Mon, Feb 19, 2024, at 6:47 AM, Peter Eisentraut wrote: > Some review of the v21 patch: Thanks for checking. > - commit message > > Mention pg_createsubscriber in the commit message title. That's the > most important thing that someone doing git log searches in the future > will be looking

Re: Injection points: some tools to wait and wake

2024-02-19 Thread Michael Paquier
On Mon, Feb 19, 2024 at 02:28:04PM +, Bertrand Drouvot wrote: > +CREATE FUNCTION injection_points_wake() > > what about injection_points_wakeup() instead? Sure. > +/* Shared state information for injection points. */ > +typedef struct InjectionPointSharedState > +{ > + /* protects

Re: Injection points: some tools to wait and wake

2024-02-19 Thread Michael Paquier
On Mon, Feb 19, 2024 at 11:54:20AM +0300, Andrey M. Borodin wrote: > 1. injection_points_wake() will wake all of waiters. But it's not > suitable for complex tests. I think there must be a way to wake only > specific waiter by injection point name. I don't disagree with that, but I don't have a

Re: 035_standby_logical_decoding unbounded hang

2024-02-19 Thread Noah Misch
On Fri, Feb 16, 2024 at 06:37:38AM +, Bertrand Drouvot wrote: > On Thu, Feb 15, 2024 at 12:48:16PM -0800, Noah Misch wrote: > > On Wed, Feb 14, 2024 at 03:31:16PM +, Bertrand Drouvot wrote: > > > What about creating a sub, say wait_for_restart_lsn_calculation() in > > > Cluster.pm > > >

Re: Avoid switching between system-user and system-username in the doc

2024-02-19 Thread Michael Paquier
On Mon, Feb 19, 2024 at 06:00:11PM +0100, Jelte Fennema-Nio wrote: > On Mon, 19 Feb 2024 at 09:52, Bertrand Drouvot > wrote: >> Please find attached a tiny patch to clean that up. > > LGTM Looks like a mistake from me in efb6f4a4f9b6, will fix and backpatch for consistency. -- Michael

Re: Streaming read-ready sequential scan code

2024-02-19 Thread Melanie Plageman
On Mon, Jan 29, 2024 at 4:17 PM Melanie Plageman wrote: > > There is an outstanding question about where to allocate the > PgStreamingRead object for sequential scans I've written three alternative implementations of the actual streaming read user for sequential scan which handle the question of

Re: Possible to trigger autovacuum?

2024-02-19 Thread Michael Paquier
On Mon, Feb 19, 2024 at 03:15:29PM -0600, Chris Cleveland wrote: > Is it possible to launch an autovacuum from within an extension? > > I'm developing an index access method. After the index gets built it needs > some cleanup and optimization. I'd prefer to do this in the > amvacuumcleanup()

Re: Patch: Add parse_type Function

2024-02-19 Thread David E. Wheeler
On Feb 19, 2024, at 15:47, Tom Lane wrote: >> 1. Add a to_regtypmod() for those who just want the typemod. > > Seems like there's a good case for doing that. I’ll work on that. > I'm less thrilled about that, mainly because I can't think of > a good name for it ("format_type_string" is

Re: Add last_commit_lsn to pg_stat_database

2024-02-19 Thread Michael Paquier
On Mon, Feb 19, 2024 at 10:26:43AM +0100, Tomas Vondra wrote: > On 2/19/24 07:57, Michael Paquier wrote: > > On Sun, Feb 18, 2024 at 02:28:06AM +0100, Tomas Vondra wrote: >>> 1) Do we really need to modify RecordTransactionCommitPrepared and >>> XactLogCommitRecord to return the LSN of the commit

Re: Patch: Add parse_type Function

2024-02-19 Thread Tom Lane
I wrote: > I'm less thrilled about that, mainly because I can't think of > a good name for it ("format_type_string" is certainly not that). After some time ruminating, a couple of possibilities occurred to me: reformat_type(text) canonical_type_name(text) > Is the use-case for

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tom Lane
Tomas Vondra writes: > On 2/19/24 16:45, Tom Lane wrote: >> Tomas Vondra writes: >>> For example, I don't think we expect selectivity functions to allocate >>> long-lived objects, right? So maybe we could run them in a dedicated >>> memory context, and reset it aggressively (after each call).

Possible to trigger autovacuum?

2024-02-19 Thread Chris Cleveland
Is it possible to launch an autovacuum from within an extension? I'm developing an index access method. After the index gets built it needs some cleanup and optimization. I'd prefer to do this in the amvacuumcleanup() method so it can happen periodically and asynchronously. I could fire up a

Re: Patch: Add parse_type Function

2024-02-19 Thread Tom Lane
"David E. Wheeler" writes: > The only way I can think of to avoid that is to: > 1. Add a to_regtypmod() for those who just want the typemod. Seems like there's a good case for doing that. > 2. Add a format_type_string() function that returns a string, the equivalent > of this function but in

Re: Why is pq_begintypsend so slow?

2024-02-19 Thread Andres Freund
Hi, On 2024-02-19 10:02:52 +0900, Sutou Kouhei wrote: > In <20240218200906.zvihkrs46yl2j...@awork3.anarazel.de> > "Re: Why is pq_begintypsend so slow?" on Sun, 18 Feb 2024 12:09:06 -0800, > Andres Freund wrote: > > >> [1] > >>

Re: PGC_SIGHUP shared_buffers?

2024-02-19 Thread Andres Freund
Hi, On 2024-02-19 13:54:01 -0500, Joe Conway wrote: > On 2/19/24 13:13, Andres Freund wrote: > > On 2024-02-19 09:19:16 -0500, Joe Conway wrote: > > > Couldn't we scale the rounding, e.g. allow small allocations as we do now, > > > but above some number always round? E.g. maybe >= 2GB round to

Re: Proposal: Adjacent B-Tree index

2024-02-19 Thread Matthias van de Meent
On Mon, 19 Feb 2024 at 18:48, Dilshod Urazov wrote: > > - Motivation > > A regular B-tree index provides efficient mapping of key values to tuples > within a table. However, if you have two tables connected in some way, a > regular B-tree index may not be efficient enough. In this case, you

Re: PGC_SIGHUP shared_buffers?

2024-02-19 Thread Joe Conway
On 2/19/24 13:13, Andres Freund wrote: On 2024-02-19 09:19:16 -0500, Joe Conway wrote: Couldn't we scale the rounding, e.g. allow small allocations as we do now, but above some number always round? E.g. maybe >= 2GB round to the nearest 256MB, >= 4GB round to the nearest 512MB, >= 8GB round to

Re: PGC_SIGHUP shared_buffers?

2024-02-19 Thread Andres Freund
Hi, On 2024-02-19 09:19:16 -0500, Joe Conway wrote: > On 2/18/24 15:35, Andres Freund wrote: > > On 2024-02-18 17:06:09 +0530, Robert Haas wrote: > > > How many people set shared_buffers to something that's not a whole > > > number of GB these days? > > > > I'd say the vast majority of postgres

Proposal: Adjacent B-Tree index

2024-02-19 Thread Dilshod Urazov
- Motivation A regular B-tree index provides efficient mapping of key values to tuples within a table. However, if you have two tables connected in some way, a regular B-tree index may not be efficient enough. In this case, you would need to create an index for each table. The purpose will become

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tomas Vondra
On 2/19/24 16:45, Tom Lane wrote: > Tomas Vondra writes: >> Considering there are now multiple patches improving memory usage during >> planning with partitions, perhaps it's time to take a step back and >> think about how we manage (or rather not manage) memory during query >> planning, and see

Re: Avoid switching between system-user and system-username in the doc

2024-02-19 Thread Jelte Fennema-Nio
On Mon, 19 Feb 2024 at 09:52, Bertrand Drouvot wrote: > Please find attached a tiny patch to clean that up. LGTM

Re: Patch: Add parse_type Function

2024-02-19 Thread David E. Wheeler
On Feb 18, 2024, at 15:55, Erik Wienhold wrote: >> The overhead of parse_type_and_format can be related to higher planning >> time. PL/pgSQL can assign composite without usage FROM clause. > > Thanks, didn't know that this makes a difference. In that case both > variants are on par.

Re: JIT compilation per plan node

2024-02-19 Thread Tomas Vondra
Hi Melih, On 1/2/24 20:50, Melih Mutlu wrote: > Hi hackers, > > After discussing this with David offlist, I decided to reinitiate this > discussion that has already been raised and discussed several times in the > past. [1] [2] > > Currently, if JIT is enabled, the decision for JIT compilation

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tom Lane
Tomas Vondra writes: > Considering there are now multiple patches improving memory usage during > planning with partitions, perhaps it's time to take a step back and > think about how we manage (or rather not manage) memory during query > planning, and see if we could improve that instead of an

Re: numeric_big in make check?

2024-02-19 Thread Tom Lane
Dean Rasheed writes: > On 19 Feb 2024, at 12:48, Tom Lane wrote: >> Or we could just flush it. It's never detected a bug, and I think >> you'd find that it adds zero code coverage (or if not, we could >> fix that in a far more surgical and less expensive manner). > Off the top of my head, I

Re: Add trim_trailing_whitespace to editorconfig file

2024-02-19 Thread Jelte Fennema-Nio
On Fri, 16 Feb 2024 at 11:45, Peter Eisentraut wrote: > I have committed that one. Thanks :) > v3-0002-Require-final-newline-in-.po-files.patch > > The .po files are imported from elsewhere, so I'm not sure this is going > to have the desired effect. Perhaps it's worth cleaning up, but it >

Use streaming read API in ANALYZE

2024-02-19 Thread Nazir Bilal Yavuz
Hi, I worked on using the currently proposed streaming read API [1] in ANALYZE. The patch is attached. 0001 is the not yet merged streaming read API code changes that can be applied to the master, 0002 is the actual code. The blocks to analyze are obtained by using the streaming read API now. -

Re: partitioning and identity column

2024-02-19 Thread Alexander Lakhin
Hello Ashutosh, 19.02.2024 15:17, Ashutosh Bapat wrote: Functions ATExecAddIdentity() and ATExecDropIdentity() are recursive too, so I think they can be exploited as well. not just Identity related functions, but many other functions in tablecmds.c have that problem as I mentioned earlier.

Re: Switching XLog source from archive to streaming when primary available

2024-02-19 Thread Japin Li
On Mon, 19 Feb 2024 at 18:36, Bharath Rupireddy wrote: > On Wed, Jan 31, 2024 at 6:30 PM Bharath Rupireddy > wrote: >> >> Needed a rebase due to commit 776621a (conflict in >> src/test/recovery/meson.build for new TAP test file added). Please >> find the attached v17 patch. > > Strengthened

Re: Injection points: some tools to wait and wake

2024-02-19 Thread Bertrand Drouvot
Hi, On Mon, Feb 19, 2024 at 04:51:45PM +0900, Michael Paquier wrote: > On Mon, Feb 19, 2024 at 03:01:40PM +0900, Michael Paquier wrote: > > 0002 is a polished version of the TAP test that makes use of this > > facility, providing coverage for the bug fixed by 7863ee4def65 > > (reverting this

Re: PGC_SIGHUP shared_buffers?

2024-02-19 Thread Joe Conway
On 2/18/24 15:35, Andres Freund wrote: On 2024-02-18 17:06:09 +0530, Robert Haas wrote: How many people set shared_buffers to something that's not a whole number of GB these days? I'd say the vast majority of postgres instances in production run with less than 1GB of s_b. Just because numbers

Re: Speeding up COPY TO for uuids and arrays

2024-02-19 Thread Laurenz Albe
On Sat, 2024-02-17 at 12:24 -0800, Andres Freund wrote: > On 2024-02-17 17:48:23 +0100, Laurenz Albe wrote: > > - Patch 0001 speeds up pq_begintypsend with a custom macro. > > That brought the binary copy down to 3.7 seconds, which is a > > speed gain of 15%. > > Nice win, but I think we can

Re: numeric_big in make check?

2024-02-19 Thread Dean Rasheed
> > On 19 Feb 2024, at 12:48, Tom Lane wrote: > > > > Or we could just flush it. It's never detected a bug, and I think > > you'd find that it adds zero code coverage (or if not, we could > > fix that in a far more surgical and less expensive manner). > Off the top of my head, I can't say to

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tomas Vondra
On 9/8/23 07:11, Lepikhov Andrei wrote: > > > On Wed, Sep 6, 2023, at 8:09 PM, Ashutosh Bapat wrote: >> Hi Lepikhov, >> >> Thanks for using my patch and I am glad that you found it useful. >> >> On Mon, Sep 4, 2023 at 10:56 AM Lepikhov Andrei >> wrote: >>> >>> Hi, hackers, >>> >>> Looking at

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-02-19 Thread Ashutosh Bapat
On Mon, Feb 19, 2024 at 4:35 AM Tomas Vondra wrote: > > Hi, > > After taking a look at the patch optimizing SpecialJoinInfo allocations, > I decided to take a quick look at this one too. I don't have any > specific comments on the code yet, but it seems quite a bit more complex > than the other

Re: Experiments with Postgres and SSL

2024-02-19 Thread Matthias van de Meent
I've been asked to take a look at this thread and review some patches, and the subject looks interesting enough, so here I am. On Thu, 19 Jan 2023 at 04:16, Greg Stark wrote: > I had a conversation a while back with Heikki where he expressed that > it was annoying that we negotiate SSL/TLS the

Re: serial not accepted as datatype in ALTER TABLE ... ALTER COLUMN

2024-02-19 Thread Ashutosh Bapat
On Sun, Feb 18, 2024 at 1:59 PM Andy Fan wrote: > > > Hi Ashutosh, > > > data_type is described on that page as "Data type of the new column, > > or new data type for an existing column." but CREATE TABLE > > documentation [2] redirects data_type to [3], which mentions serial. > > The impression

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-19 Thread Ashutosh Bapat
On Sun, Feb 18, 2024 at 10:55 PM Tomas Vondra wrote: > > Hi, > > I took a quick look at this patch today. I certainly agree with the > intent to reduce the amount of memory during planning, assuming it's not > overly disruptive. And I think this patch is fairly localized and looks > sensible.

Re: Synchronizing slots from primary to standby

2024-02-19 Thread shveta malik
On Mon, Feb 19, 2024 at 5:32 PM Amit Kapila wrote: > > Few comments on 0001 Thanks for the feedback. > > 1. I think it is better to error out when the valid GUC or option is > not set in ensure_valid_slotsync_params() and > ensure_valid_remote_info() instead of waiting. And

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Ranier Vilela
Em seg., 19 de fev. de 2024 às 05:35, Andrei Lepikhov < a.lepik...@postgrespro.ru> escreveu: > On 16/2/2024 19:54, jian he wrote: > > After setting these parameters, overall enable_or_transformation ON is > > performance better. > > sorry for the noise. > Don't worry, at least we know a weak

Re: Memory consumed by paths during partitionwise join planning

2024-02-19 Thread Ashutosh Bapat
On Fri, Feb 16, 2024 at 8:42 AM Andrei Lepikhov wrote: > Live example: right now, I am working on the code like MSSQL has - a > combination of NestLoop and HashJoin paths and switching between them in > real-time. It requires both paths in the path list at the moment when > extensions are coming.

Re: partitioning and identity column

2024-02-19 Thread Ashutosh Bapat
On Thu, Feb 15, 2024 at 11:30 PM Alexander Lakhin wrote: > > Hello Ashutosh, > > 24.01.2024 09:34, Ashutosh Bapat wrote: > > > >>> There's another thing I found. The file isn't using > >>> check_stack_depth() in the function which traverse inheritance > >>> hierarchies. This isn't just a problem

Re: Synchronizing slots from primary to standby

2024-02-19 Thread Amit Kapila
On Mon, Feb 19, 2024 at 9:46 AM shveta malik wrote: > > Okay I see. Thanks for pointing it out. Here are the patches > addressing your comments. Changes are in patch001, rest are rebased. > Few comments on 0001 1. I think it is better to error out when the valid GUC or

Re: numeric_big in make check?

2024-02-19 Thread Tom Lane
Daniel Gustafsson writes: > numeric_big has been left out of parallel_schedule, requiring EXTRA_TESTS to > run it, since going in back in 1999 (AFAICT it was even the reason EXTRA_TESTS > was invented). The original commit states that it's huge, and it probably > was. > Today it runs faster

Re: Add an option to skip loading missing publication to avoid logical replication failure

2024-02-19 Thread vignesh C
On Mon, 19 Feb 2024 at 12:48, vignesh C wrote: > > Hi, > > Currently ALTER SUBSCRIPTION ... SET PUBLICATION will break the > logical replication in certain cases. This can happen as the apply > worker will get restarted after SET PUBLICATION, the apply worker will > use the existing slot and

Re: table inheritance versus column compression and storage settings

2024-02-19 Thread Ashutosh Bapat
On Fri, Feb 16, 2024 at 11:54 PM Tom Lane wrote: > > I wrote: > > I find it surprising that the committed patch does not touch > > pg_dump. Is it really true that pg_dump dumps situations with > > differing compression/storage settings accurately already? > > It's worse than I thought. Run

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-19 Thread John Naylor
On Mon, Feb 19, 2024 at 9:02 AM Masahiko Sawada wrote: > > I think that vacuum and tidbitmap (and future users) would end up > having the same max block size calculation. And it seems slightly odd > layering to me that max-block-size-specified context is created on > vacuum (or tidbitmap) layer,

Re: Switching XLog source from archive to streaming when primary available

2024-02-19 Thread Bharath Rupireddy
On Wed, Jan 31, 2024 at 6:30 PM Bharath Rupireddy wrote: > > Needed a rebase due to commit 776621a (conflict in > src/test/recovery/meson.build for new TAP test file added). Please > find the attached v17 patch. Strengthened tests a bit by using recovery_min_apply_delay to mimic standby spending

Re: speed up a logical replica setup

2024-02-19 Thread Shlok Kyal
Hi, I have reviewed the v21 patch. And found an issue. Initially I started the standby server with a new postgresql.conf file (not the default postgresql.conf that is present in the instance). pg_ctl -D ../standby start -o "-c config_file=/new_path/postgresql.conf" And I have made

Re: Transaction timeout

2024-02-19 Thread Japin Li
On Mon, 19 Feb 2024 at 17:14, Andrey M. Borodin wrote: >> On 18 Feb 2024, at 22:16, Andrey M. Borodin wrote: >> >> But it seems a little strange that session 3 did not fail at all > It was only coincidence. Any test that verifies FATALing out in 100ms can > fail, see new failure here [0]. >

Re: Fix race condition in InvalidatePossiblyObsoleteSlot()

2024-02-19 Thread Bertrand Drouvot
Hi, On Mon, Feb 19, 2024 at 01:45:16PM +0530, Bharath Rupireddy wrote: > On Mon, Feb 19, 2024 at 11:44 AM Michael Paquier wrote: > > > > > Yeah, comments added in v3. > > > > The contents look rather OK, I may do some word-smithing for both. > > Here are some comments on v3: Thanks for looing

Re: speed up a logical replica setup

2024-02-19 Thread Peter Eisentraut
Some review of the v21 patch: - commit message Mention pg_createsubscriber in the commit message title. That's the most important thing that someone doing git log searches in the future will be looking for. - doc/src/sgml/ref/allfiles.sgml Move the new entry to alphabetical order. -

numeric_big in make check?

2024-02-19 Thread Daniel Gustafsson
numeric_big has been left out of parallel_schedule, requiring EXTRA_TESTS to run it, since going in back in 1999 (AFAICT it was even the reason EXTRA_TESTS was invented). The original commit states that it's huge, and it probably was. Today it runs faster than many tests we have in

Re: Add last_commit_lsn to pg_stat_database

2024-02-19 Thread Tomas Vondra
On 2/19/24 07:57, Michael Paquier wrote: > On Sun, Feb 18, 2024 at 02:28:06AM +0100, Tomas Vondra wrote: >> Thanks for the updated patch. I don't have a clear opinion on the >> feature and whether this is the way to implement it, but I have two >> simple questions. > > Some users I know of would

Re: Transaction timeout

2024-02-19 Thread Andrey M. Borodin
> On 18 Feb 2024, at 22:16, Andrey M. Borodin wrote: > > But it seems a little strange that session 3 did not fail at all It was only coincidence. Any test that verifies FATALing out in 100ms can fail, see new failure here [0]. In a nearby thread Michael is proposing injections points that

Re: System username in pg_stat_activity

2024-02-19 Thread Bertrand Drouvot
Hi, On Fri, Feb 16, 2024 at 09:41:41PM +0100, Magnus Hagander wrote: > On Fri, Feb 16, 2024 at 8:55 PM Andres Freund wrote: > > > > Hi, > > > > On 2024-01-12 17:16:53 +0100, Magnus Hagander wrote: > > > On Thu, Jan 11, 2024 at 5:55 PM Bertrand Drouvot > > > wrote: > > > > On Thu, Jan 11, 2024

  1   2   >