Re: Support logical replication of DDLs

2023-07-17 Thread Michael Paquier
On Tue, Jul 18, 2023 at 02:28:08PM +0900, Masahiko Sawada wrote: > I've considered some alternative approaches but I prefer the second > approach. A long test time could not be a big problem unless we run it > by default. We can prepare a buildfarm animal that is configured to > run the DDL

Re: Add TOAST support for more system tables

2023-07-17 Thread Michael Paquier
On Mon, Jul 17, 2023 at 06:31:04PM -0400, Tom Lane wrote: > Sofia Kopikova writes: > > This patch adds TOAST support for system tables pg_class, > > pg_attribute and pg_largeobject_metadata, as they include ACL columns, > > which may be potentially large in size. > > We have been around on this

Re: Support logical replication of DDLs

2023-07-17 Thread Masahiko Sawada
On Tue, Jul 11, 2023 at 8:01 PM Zhijie Hou (Fujitsu) wrote: > > Hi, > > We have been researching how to create a test that detects failures resulting > from future syntax changes, where the deparser fails to update properly. > > The basic idea comes from what Robert Haas suggested in [1]: when

Re: logicalrep_message_type throws an error

2023-07-17 Thread Masahiko Sawada
On Tue, Jul 18, 2023 at 12:15 PM Amit Kapila wrote: > > On Mon, Jul 17, 2023 at 7:54 PM Alvaro Herrera > wrote: > > > > On 2023-Jul-17, Ashutosh Bapat wrote: > > > > > Prologue of psprintf() says > > > > > > * Errors are not returned to the caller, but are reported via elog(ERROR) > > > * in

Re: Inefficiency in parallel pg_restore with many tables

2023-07-17 Thread Nathan Bossart
On Sun, Jul 16, 2023 at 08:54:24PM -0700, Nathan Bossart wrote: > This seems worth a try. IIUC you are suggesting making binaryheap.c > frontend-friendly and expanding its API a bit. If no one has volunteered, > I could probably hack something together. I spent some time on the binaryheap

Volatile write caches on macOS and Windows, redux

2023-07-17 Thread Thomas Munro
Hi, Continuing a topic from earlier threads[1][2], I've been wondering about how to de-klugify wal_sync_method=fsync_writethrough (a setting that actually affects much more than just WAL), and how to do the right thing for our users on macOS and Windows by default. Commit d0c28601 was a very

Re: logicalrep_message_type throws an error

2023-07-17 Thread Amit Kapila
On Mon, Jul 17, 2023 at 7:54 PM Alvaro Herrera wrote: > > On 2023-Jul-17, Ashutosh Bapat wrote: > > > Prologue of psprintf() says > > > > * Errors are not returned to the caller, but are reported via elog(ERROR) > > * in the backend, or printf-to-stderr-and-exit() in frontend builds. > > * One

RE: Partial aggregates pushdown

2023-07-17 Thread fujii.y...@df.mitsubishielectric.co.jp
Hi Mr.Pyhalov. > From: Alexander Pyhalov > Sent: Friday, July 14, 2023 10:40 PM > 1) In foreign_join_ok() should we set fpinfo->user if > fpinfo->check_partial_aggregate_support is set like it's done for > fpinfo->use_remote_estimate? It seems we can end up with fpinfo->user > fpinfo->= > NULL

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-17 Thread Peter Smith
On Tue, Jul 18, 2023 at 11:25 AM Peter Smith wrote: > > On Tue, Jul 18, 2023 at 1:54 AM Melih Mutlu wrote: > > > > Hi, > > > > PFA updated patches. Rebased 0003 with minor changes. Addressed Peter's > > reviews for 0001 and 0002 with some small comments below. > > > > Thanks, I will take

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-17 Thread Peter Smith
On Tue, Jul 18, 2023 at 1:54 AM Melih Mutlu wrote: > > Hi, > > PFA updated patches. Rebased 0003 with minor changes. Addressed Peter's > reviews for 0001 and 0002 with some small comments below. > Thanks, I will take another look at these soon. FYI, the 0001 patch does not apply cleanly. It

Re: CommandStatus from insert returning when using a portal.

2023-07-17 Thread David G. Johnston
On Wed, Jul 12, 2023 at 2:49 PM Tom Lane wrote: > Dave Cramer writes: > > Obviously I am biased by the JDBC API which would like to have > > PreparedStatement.execute() return the number of rows inserted > > without having to wait to read all of the rows returned > > Umm ... you do realize that

Re: Dumping policy on a table belonging to an extension is expected?

2023-07-17 Thread Stephen Frost
Greetings, * Amul Sul (sula...@gmail.com) wrote: > I have a ROW LEVEL SECURITY policy on the table part of an extension, and > while > dumping the database where that extension is installed, dumps the policy of > that table too even though not dumpling that table . > > Here is quick tests, where

Re: Atomic ops for unlogged LSN

2023-07-17 Thread Andres Freund
Hi, On 2023-07-17 16:15:52 -0700, Nathan Bossart wrote: > On Mon, Jul 17, 2023 at 07:08:03PM -0400, Stephen Frost wrote: > > Awesome. Was there any other feedback on this change which basically is > > just getting rid of a spinlock and replacing it with using atomics? > > It's still in

Re: Add TOAST support for more system tables

2023-07-17 Thread David Rowley
On Tue, 18 Jul 2023 at 10:31, Tom Lane wrote: > I wonder whether we'd be better off shoving the ACL data out of > these catalogs and putting it somewhere else (compare pg_attrdef). relpartbound is another column that could cause a pg_class row to grow too large. I did have a patch [1] to move

Re: unrecognized node type while displaying a Path due to dangling pointer

2023-07-17 Thread David Rowley
On Mon, 17 Jul 2023 at 15:31, Tom Lane wrote: > > I also didn't do anything about ExtensibleNode types. I assume just > > copying the ExtensibleNode isn't good enough. To flat copy the actual > > node I think would require adding a new function to > > ExtensibleNodeMethods. > > Yeah, the problem

Re: Atomic ops for unlogged LSN

2023-07-17 Thread Nathan Bossart
On Mon, Jul 17, 2023 at 07:08:03PM -0400, Stephen Frost wrote: > Awesome. Was there any other feedback on this change which basically is > just getting rid of a spinlock and replacing it with using atomics? > It's still in needs-review status but there's been a number of > comments/reviews

Re: Atomic ops for unlogged LSN

2023-07-17 Thread Stephen Frost
Greetings, * Nathan Bossart (nathandboss...@gmail.com) wrote: > On Mon, Jun 12, 2023 at 07:24:18PM -0400, Stephen Frost wrote: > > * Nathan Bossart (nathandboss...@gmail.com) wrote: > >> Is it? I see uses in GiST indexing (62401db), so it's not immediately > >> obvious to me how it is

Re: Add TOAST support for more system tables

2023-07-17 Thread Tom Lane
Sofia Kopikova writes: > This patch adds TOAST support for system tables pg_class, > pg_attribute and pg_largeobject_metadata, as they include ACL columns, > which may be potentially large in size. We have been around on this topic before, cf discussion leading up to commit 96cdeae07. Allowing

Re: ObjectIdGetDatum() missing from SearchSysCache*() callers

2023-07-17 Thread Michael Paquier
On Mon, Jul 17, 2023 at 05:33:42PM +0300, Aleksander Alekseev wrote: > I can't be 100% sure but it looks like that's all of them. PFA the > updated patch v2. Thanks. Yes, this stuff is easy to miss. I was just grepping for a few patterns and missed these two. -- Michael signature.asc

Add TOAST support for more system tables

2023-07-17 Thread Sofia Kopikova
Hi, hackers! I've tried sending this patch to community before, let me try it second time. Patch is revised and improved compared to previous version. This patch adds TOAST support for system tables pg_class, pg_attribute and pg_largeobject_metadata, as they include ACL columns, which may be

Re: harmonize password reuse in vacuumdb, clusterdb, and reindexdb

2023-07-17 Thread Nathan Bossart
On Wed, Jun 28, 2023 at 10:24:09PM -0700, Nathan Bossart wrote: > On Wed, Jun 28, 2023 at 09:20:03PM -0700, Gurjeet Singh wrote: >> The comment on top of connect_utils.c:connectDatabase() seems pertinent: >> >>> (Callers should not pass >>> * allow_password_reuse=true unless reconnecting to the

Re: SLRUs in the main buffer pool - Page Header definitions

2023-07-17 Thread Stephen Frost
Greetings, [snipped quoted bits] Would really be helpful to keep who the author of each quoted snipper was when you quote them; dropping that makes it look like one person wrote all of them and that's confusing. * Bagga, Rishu (bagri...@amazon.com) wrote: > The third patch brings back the the

Re: Fine-tune TLS 1.3 cipher suites and curves lists

2023-07-17 Thread Daniel Gustafsson
> On 17 Jul 2023, at 15:16, Seraphime Kirkovski wrote: > I see in the source code that only TLS 1.2 and bellow cipher lists can be > configured: > > https://github.com/postgres/postgres/blob/master/src/backend/libpq/be-secure-openssl.c#L281 > > and Postgres relies on the OpenSSL defaults

Re: MERGE ... RETURNING

2023-07-17 Thread Jeff Davis
On Fri, 2023-07-14 at 09:55 +0100, Dean Rasheed wrote: > I found a 10-year-old thread discussing adding support for OLD/NEW to > RETURNING [1], but it doesn't look like anything close to a > committable solution was developed, or even a design that might lead > to one. That's a shame, because

Re: Fix search_path for all maintenance commands

2023-07-17 Thread Jeff Davis
On Mon, 2023-07-17 at 10:58 -0700, Nathan Bossart wrote: > On Sat, Jul 15, 2023 at 02:13:33PM -0700, Noah Misch wrote: > > The 2018 security fixes instigated many function repairs that > > $SUBJECT would > > otherwise instigate.  That wasn't too painful.  The net new pain of > > $SUBJECT > > will

Re: Should we remove db_user_namespace?

2023-07-17 Thread Nathan Bossart
On Sun, Jul 16, 2023 at 01:24:06PM +0200, Magnus Hagander wrote: > I'd lean towards "no". A hard break, when it's a major release, is > better than a "it stopped having effect but didn't tell you anything" > break. Especially when it comes to things like startup scripts etc. Committed. --

Re: sslinfo extension - add notbefore and notafter timestamps

2023-07-17 Thread Cary Huang
Hello > > Perhaps calling "tm2timestamp(_time, 0, NULL, )" without checking > > the return code would be just fine. I see some other usages of > > tm2timstamp() in other code areas also skip checking the return code. > > I think we want to know about any failures, btu we can probably make

Re: Looking for context around which event triggers are permitted

2023-07-17 Thread Garrett Thornburg
That makes sense and is similar to the problem I'm hoping to solve for our team. We had a DB upgrade that corrupted a few indexes. Gitlab went through something similar as part of their OS/ DB upgrade. We had to concurrently reindex everything. This took a few days and just to make sure we

Re: Fix search_path for all maintenance commands

2023-07-17 Thread Nathan Bossart
On Sat, Jul 15, 2023 at 02:13:33PM -0700, Noah Misch wrote: > The 2018 security fixes instigated many function repairs that $SUBJECT would > otherwise instigate. That wasn't too painful. The net new pain of $SUBJECT > will be less, since the 2018 security fixes prepared the path. Hence, I >

Report distinct wait events when waiting for WAL "operation"

2023-07-17 Thread Andres Freund
Hi, In a number of workloads one can see two wait events prominently: LWLock:WALWrite and LWLock:WALInsert. Unfortunately for both that is not very informative: LWLock:WALWrite can be reported because there is genuine contention on the LWLock, or, more commonly, because several backends are

Re: generic plans and "initial" pruning

2023-07-17 Thread Thom Brown
On Thu, 13 Jul 2023 at 13:59, Amit Langote wrote: > In an absolutely brown-paper-bag moment, I realized that I had not > updated src/backend/executor/README to reflect the changes to the > executor's control flow that this patch makes. That is, after > scrapping the old design back in January

Re: Looking for context around which event triggers are permitted

2023-07-17 Thread Aleksander Alekseev
Hi, > Happy to enable those with a patch if it's as Aleksander said. Meaning, no > real reason they were disabled other than someone thought folks wouldn't need > them. Sure, please feel free submitting the patch and we will see how it goes. I don't foresee a strong push-back from the

Re: Looking for context around which event triggers are permitted

2023-07-17 Thread Alvaro Herrera
On 2023-Jul-17, Garrett Thornburg wrote: > That's a good point, Isaac. Select into, security label, comment, etc are > all maintenance style commands but are already added to the matrix. I do > think there's a good case to include other maintenance related commands as > event triggers. Suppose

Re: Looking for context around which event triggers are permitted

2023-07-17 Thread Garrett Thornburg
That's a good point, Isaac. Select into, security label, comment, etc are all maintenance style commands but are already added to the matrix. I do think there's a good case to include other maintenance related commands as event triggers. Suppose you want to know the last time a table was vacuumed

Re: Looking for context around which event triggers are permitted

2023-07-17 Thread Isaac Morland
On Mon, 17 Jul 2023 at 11:26, Aleksander Alekseev wrote: > Hi, > > > I was working on a project with event triggers and was wondering if > there was any context from the developers around why some things make this > list and others do not. Example: REVOKE/ GRANT are in the event trigger > matrix

Re: Looking for context around which event triggers are permitted

2023-07-17 Thread Aleksander Alekseev
Hi, > I was working on a project with event triggers and was wondering if there was > any context from the developers around why some things make this list and > others do not. Example: REVOKE/ GRANT are in the event trigger matrix [1] but > REINDEX is not. Just wondering if there's a mailing

Fine-tune TLS 1.3 cipher suites and curves lists

2023-07-17 Thread Seraphime Kirkovski
Hi all, I’m a security engineer and I’m looking into restricting the set of allowed ciphers on Postgres and configure a concrete set of curves on our postgres instances. I see in the source code that only TLS 1.2 and bellow cipher lists can be configured:

Re: Improve heapgetpage() performance, overhead from serializable

2023-07-17 Thread Andres Freund
Hi, On 2023-07-17 09:55:07 +0800, Zhang Mingli wrote: > LGTM and I have a fool question: > > if (likely(all_visible)) > { > if (likely(!check_serializable)) > scan->rs_ntuples = heapgetpage_collect(scan, snapshot, > page, buffer, >

Looking for context around which event triggers are permitted

2023-07-17 Thread Garrett Thornburg
Hey list, I was working on a project with event triggers and was wondering if there was any context from the developers around why some things make this list and others do not. Example: REVOKE/ GRANT are in the event trigger matrix [1] but REINDEX is not. Just wondering if there's a mailing list

Re: ObjectIdGetDatum() missing from SearchSysCache*() callers

2023-07-17 Thread Aleksander Alekseev
Hi, > > And inside pg_sequence_parameters: > > pgstuple = SearchSysCache1(SEQRELID, relid); > > Found another one in partcache.c: > > ``` > /* Get pg_class.relpartbound */ > tuple = SearchSysCache1(RELOID, RelationGetRelid(rel)); > ``` > > I can't be 100% sure but it looks like that's all

Re: ObjectIdGetDatum() missing from SearchSysCache*() callers

2023-07-17 Thread Aleksander Alekseev
Hi, > And inside pg_sequence_parameters: > pgstuple = SearchSysCache1(SEQRELID, relid); Found another one in partcache.c: ``` /* Get pg_class.relpartbound */ tuple = SearchSysCache1(RELOID, RelationGetRelid(rel)); ``` I can't be 100% sure but it looks like that's all of them. PFA the

Re: [PATCH] Automatic HASH and LIST partition creation

2023-07-17 Thread Stéphane Tachoires
Hi, I found that thread (and the patch), but it seems to be pretty dead. Patch didn't apply, due to gen_node_support.pl Can I hope for a rebirth ? I've made a rebased patch,in case of no response... It's just the patch from

Re: logicalrep_message_type throws an error

2023-07-17 Thread Alvaro Herrera
On 2023-Jul-17, Ashutosh Bapat wrote: > Prologue of psprintf() says > > * Errors are not returned to the caller, but are reported via elog(ERROR) > * in the backend, or printf-to-stderr-and-exit() in frontend builds. > * One should therefore think twice about using this in libpq. > > If an

Re: Getting rid of OverrideSearhPath in namespace.c

2023-07-17 Thread Aleksander Alekseev
Hi, > As a follow-up for the CVE-2023-2454 fix, I think that it makes sense to > completely remove unsafe functions > PushOverrideSearchPath()/PopOverrideSearchPath(), which are not used in the > core now. > Please look at the patch attached. > > [...] > > What do you think? +1 to remove dead

Re: logicalrep_message_type throws an error

2023-07-17 Thread Ashutosh Bapat
On Sat, Jul 15, 2023 at 12:57 PM Amit Kapila wrote: > > > > Since the numerical value is important only in invalid message type > > cases, how about using a format like "??? (88)" in unknown message > > type cases, in both error and context messages? > > > > Do you have something like attached in

Re: ObjectIdGetDatum() missing from SearchSysCache*() callers

2023-07-17 Thread Zhang Mingli
Hi, Regards, Zhang Mingli On Jul 17, 2023 at 21:09 +0800, Zhang Mingli , wrote: > sequence_options And inside pg_sequence_parameters: pgstuple = SearchSysCache1(SEQRELID, relid);

Re: ObjectIdGetDatum() missing from SearchSysCache*() callers

2023-07-17 Thread Zhang Mingli
Hi Regards, Zhang Mingli On Jul 17, 2023 at 19:10 +0800, Michael Paquier , wrote: > Hi all, > > While scanning the code, I have noticed that a couple of code paths > that do syscache lookups are passing down directly Oids rather than > Datums. I think that we'd better be consistent here, even if

Re: [EXTERNAL] Re: Add non-blocking version of PQcancel

2023-07-17 Thread Jelte Fennema
Rebased again to resolve some conflicts v22-0003-Add-non-blocking-version-of-PQcancel.patch Description: Binary data v22-0002-Return-2-from-pqReadData-on-EOF.patch Description: Binary data v22-0004-Start-using-new-libpq-cancel-APIs.patch Description: Binary data

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-07-17 Thread Amit Kapila
On Fri, Jun 30, 2023 at 7:29 PM Hayato Kuroda (Fujitsu) wrote: > > I have analyzed more, and concluded that there are no difference between > manual > and shutdown checkpoint. > > The difference was whether the CHECKPOINT record has been decoded or not. > The overall workflow of this test was: >

Re: Protect extension' internal tables - how?

2023-07-17 Thread Aleksander Alekseev
Hi, > Could you please advise or give some hint on what is the correct (and > secure) way to implement this? > > Currently I use the owner of the extension as owner when creating > such a table inside the function, but maybe there are some pitfalls > in this kind of solution? If the goal is to

Re: ObjectIdGetDatum() missing from SearchSysCache*() callers

2023-07-17 Thread Aleksander Alekseev
Hi, > I have noticed 11 callers of SearchSysCache*() that pass down > an Oid instead of a Datum. Good catch. > I think that we'd better be consistent here, even if there is > no actual bug. > +1 -- Best regards, Aleksander Alekseev

Re: Make mesage at end-of-recovery less scary.

2023-07-17 Thread Aleksander Alekseev
Hi, > Thanks for checking it! > > I think 4ac30ba4f2 is that, which changes a few error > messages. Addition to rebasing, I rewrote some code comments of > xlogreader.c and revised the additional test script. Thanks for working on this, it bugged me for a while. I noticed that cfbot is not happy

ObjectIdGetDatum() missing from SearchSysCache*() callers

2023-07-17 Thread Michael Paquier
Hi all, While scanning the code, I have noticed that a couple of code paths that do syscache lookups are passing down directly Oids rather than Datums. I think that we'd better be consistent here, even if there is no actual bug. I have noticed 11 callers of SearchSysCache*() that pass down an

Re: doc: clarify the limitation for logical replication when REPILICA IDENTITY is FULL

2023-07-17 Thread Amit Kapila
On Mon, Jul 17, 2023 at 11:51 AM Önder Kalacı wrote: > >> > >> > The last line seems repetitive to me. So, I have removed it. Apart >> > from that patch looks good to me. Sergie, Peter, and others, any >> > thoughts? >> >> The v5 patch LGTM. >> > > Overall looks good to me as well. Please

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-07-17 Thread Peter Smith
Hi Kuroda-san, I haven't looked at this thread for a very long time so to re-familiarize myself with it I read all the latest v16-0001 patch. Here are a number of minor review comments I noted in passing: == Commit message 1. For pg_dump this commit includes a new option called

Re: Some revises in adding sorting path

2023-07-17 Thread Richard Guo
On Mon, Jul 10, 2023 at 5:37 PM Daniel Gustafsson wrote: > > On 28 Mar 2023, at 21:59, David Rowley wrote: > > I'll mark this as waiting on author while you work on that. > > Richard: have you had a chance to incorporate the tests proposed by David > in > this thread into your patch? I just

Re: Some revises in adding sorting path

2023-07-17 Thread Richard Guo
On Wed, Mar 29, 2023 at 4:00 AM David Rowley wrote: > If you write some tests for this code, it will be useful to prove that > it actually does something, and also that it does not break again in > the future. I don't really want to just blindly copy the pattern used > in 3c6fc5820 for creating

Re: CommandStatus from insert returning when using a portal.

2023-07-17 Thread Alvaro Herrera
On 2023-Jul-14, Dave Cramer wrote: > David, > > I will try to get a tcpdump file. Doing this in libpq seems challenging as > I'm not aware of how to create a portal in psql. You can probably have a look at src/test/modules/libpq_pipeline/libpq_pipeline.c as a basis to write some test code for

Re: doc: improve the restriction description of using indexes on REPLICA IDENTITY FULL table.

2023-07-17 Thread Önder Kalacı
Hi, > I've attached a draft patch for this idea. I think it needs a rebase after edca3424342da323499a1998d18a888283e52ac7. Also, as discussed in [1], I think one improvement we had was to keep IsIndexUsableForReplicaIdentityFull() in a way that it is easier to read & better documented in the

Re: remaining sql/json patches

2023-07-17 Thread Erik Rijkers
Op 7/17/23 om 07:00 schreef jian he: hi. seems there is no explanation about, json_api_common_syntax in functions-json.html I can get json_query full synopsis from functions-json.html as follows: json_query ( context_item, path_expression [ PASSING { value AS varname } [, ...]] [ RETURNING

Re: doc: clarify the limitation for logical replication when REPILICA IDENTITY is FULL

2023-07-17 Thread Önder Kalacı
Hi, > > > The last line seems repetitive to me. So, I have removed it. Apart > > from that patch looks good to me. Sergie, Peter, and others, any > > thoughts? > > The v5 patch LGTM. > > Overall looks good to me as well. Please consider the following as an optional improvement. My only minor