Re: Simplify some logical replication worker type checking

2023-07-31 Thread Peter Smith
On Tue, Aug 1, 2023 at 12:59 PM Zhijie Hou (Fujitsu) wrote: > > > About 2,3,4, it seems you should use "if (am_leader_apply_worker())" instead > of > "if (!am_leader_apply_worker())" because only leader apply worker should > invoke > this function. > Hi Hou-san, Thanks for your review! Oops.

Re: Faster "SET search_path"

2023-07-31 Thread Jeff Davis
On Sat, 2023-07-29 at 12:44 -0400, Isaac Morland wrote: > Essentially, "just" observe efficiently (somehow) that no change is > needed, and skip changing it? I gave this a try and it speeds things up some more. There might be a surprise factor with an optimization like that, though. If someone

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 01:51:13PM +0900, Kyotaro Horiguchi wrote: > I believe a database server is not supposed to be executed under such > a memory-constrained environment. I don't really follow this argument. The backend and the frontends are reliable on OOM, where we generate ERRORs or even

Fix compilation warnings when CFLAGS -Og is specified

2023-07-31 Thread Hayato Kuroda (Fujitsu)
Dear hackers, # Background Based on [1], I did configure and build with options: (I used Meson build system, but it could be reproduced by Autoconf/Make) ``` $ meson setup -Dcassert=true -Ddebug=true -Dc_args=-Og ../builder $ cd ../builder $ ninja ``` My gcc version is 4.8.5, and ninja is

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Kyotaro Horiguchi
At Tue, 1 Aug 2023 12:43:21 +0900, Michael Paquier wrote in > A colleague, Ethan Mertz (in CC), has discovered that we don't handle > correctly WAL records that are failing because of an OOM when > allocating their required space. In the case of Ethan, we have bumped > on the failure after an

Extract numeric filed in JSONB more effectively

2023-07-31 Thread Andy Fan
Hi: Currently if we want to extract a numeric field in jsonb, we need to use the following expression: cast (a->>'a' as numeric). It will turn a numeric to text first and then turn the text to numeric again. See jsonb_object_field_text and JsonbValueAsText. However the binary format of numeric

Re: Extension Enhancement: Buffer Invalidation in pg_buffercache

2023-07-31 Thread Palak Chaturvedi
Hii, Thanks for your feedback. We have decided to add a role for the extension to solve that problem. And concerning to pg_unwarm table I think we can create a new function to do that but I think a general user would not require to clear a table from buffercache. We can use bufferid and where

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-31 Thread Peter Smith
On Fri, Jul 28, 2023 at 5:22 PM Peter Smith wrote: > > Hi Melih, > > BACKGROUND > -- > > We wanted to compare performance for the 2 different reuse-worker > designs, when the apply worker is already busy handling other > replications, and then simultaneously the test table tablesyncs are

Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Michael Paquier
Hi all, A colleague, Ethan Mertz (in CC), has discovered that we don't handle correctly WAL records that are failing because of an OOM when allocating their required space. In the case of Ethan, we have bumped on the failure after an allocation failure on XLogReadRecordAlloc(): "out of memory

Re: Support to define custom wait events for extensions

2023-07-31 Thread Andres Freund
Hi, On 2023-08-01 12:14:49 +0900, Michael Paquier wrote: > On Tue, Aug 01, 2023 at 11:51:35AM +0900, Masahiro Ikeda wrote: > > Thanks for committing the main patch. > > > > In my understanding, the rest works are > > * to support WaitEventExtensionMultiple() > > * to replace WAIT_EVENT_EXTENSION

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 11:51:35AM +0900, Masahiro Ikeda wrote: > Thanks for committing the main patch. > > In my understanding, the rest works are > * to support WaitEventExtensionMultiple() > * to replace WAIT_EVENT_EXTENSION to custom wait events > > Do someone already works for them? If not,

Re: logical decoding and replication of sequences, take 2

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 5:04 PM Tomas Vondra wrote: > > On 7/31/23 11:25, Amit Kapila wrote: > > On Sat, Jul 29, 2023 at 5:53 PM Tomas Vondra > > wrote: > >> > >> On 7/28/23 14:44, Ashutosh Bapat wrote: > >>> On Wed, Jul 26, 2023 at 8:48 PM Tomas Vondra > >>> wrote: > > Anyway, I was

RE: Simplify some logical replication worker type checking

2023-07-31 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 1, 2023 9:36 AM Peter Smith > PROBLEM / SOLUTION > > During recent reviews, I noticed some of these conditions are a bit unusual. Thanks for the patch. > > == > worker.c > > 1. apply_worker_exit > > /* > * Reset the last-start time for this apply worker so that the

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
On 2023-07-31 19:22, Michael Paquier wrote: I am not sure that any of that is necessary. Anyway, I have applied v11 to get the basics done. Thanks for committing the main patch. In my understanding, the rest works are * to support WaitEventExtensionMultiple() * to replace

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Julien Rouhaud
On Tue, Aug 01, 2023 at 11:37:49AM +0900, Michael Paquier wrote: > On Tue, Aug 01, 2023 at 10:22:09AM +0800, Julien Rouhaud wrote: > > Looking at the rest of the ignored patterns, the only remaining one would be > > DEALLOCATE, which AFAICS doesn't have a query_jumble_ignore tag for now. > > This

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 10:22:09AM +0800, Julien Rouhaud wrote: > Looking at the rest of the ignored patterns, the only remaining one would be > DEALLOCATE, which AFAICS doesn't have a query_jumble_ignore tag for now. This one seems to be simple as well with a location field, looking quickly at

Re: Inaccurate comments in ReorderBufferCheckMemoryLimit()

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 8:46 PM Masahiko Sawada wrote: > > While reading the code, I realized that the following code comments > might not be accurate: > > /* > * Pick the largest transaction (or subtransaction) and evict it from > * memory by streaming, if possible.

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Julien Rouhaud
On Tue, Aug 01, 2023 at 11:00:32AM +0900, Michael Paquier wrote: > On Tue, Aug 01, 2023 at 09:28:08AM +0800, Julien Rouhaud wrote: > > > FTR we had to entirely ignore all those statements in powa years ago to try > > to > > make the tool usable in such case for some users who where using 2pc, it

Re: [PATCH] Add support function for containment operators

2023-07-31 Thread Laurenz Albe
On Sat, 2023-07-08 at 08:11 +0200, Kim Johan Andersson wrote: > On 07-07-2023 13:20, Laurenz Albe wrote: > > I wrote: > > > You implement both "SupportRequestIndexCondition" and > > > "SupportRequestSimplify", > > > but when I experimented, the former was never called.  That does not > > >

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 09:28:08AM +0800, Julien Rouhaud wrote: > Having an application relying on 2pc leads to pg_stat_statements being > virtually unusable on the whole instance, so +1 for the patch. Cool, thanks for the feedback! > FTR we had to entirely ignore all those statements in powa

Simplify some logical replication worker type checking

2023-07-31 Thread Peter Smith
Hi hackers, BACKGROUND There are 3 different types of logical replication workers. 1. apply leader workers 2. parallel apply workers 3. tablesync workers And there are a number of places where the current worker type is checked using the am_ inline functions. PROBLEM / SOLUTION During recent

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Julien Rouhaud
Hi, On Tue, Aug 01, 2023 at 09:38:14AM +0900, Michael Paquier wrote: > > 31de7e6 has silenced savepoint names in the query jumbling, and > something similar can be done for 2PC transactions once the GID is > ignored in TransactionStmt. This leads to the following grouping in >

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-31 Thread Andres Freund
Hi, On 2023-07-27 20:53:16 +1200, David Rowley wrote: > To summarise, REL_15_STABLE can run this benchmark in 526.014 ms on my > AMD 3990x machine. Today's REL_16_STABLE takes 530.344 ms. We're > talking about another patch to speed up the pg_strtoint functions > which gets this down to 483.790

PostgreSQL 16 Beta 3 release date

2023-07-31 Thread Jonathan S. Katz
Hi, The release date for PostgreSQL 16 Beta 3 is August 10, 2023, alongside the regular update release[1]. Please be sure to commit any open items[2] for the Beta 3 release before August 6, 2023 0:00 AoE[3] to give them enough time to work through the buildfarm. Thanks, Jonathan [1]

Re: pg_upgrade fails with in-place tablespace

2023-07-31 Thread Michael Paquier
On Sat, Jul 29, 2023 at 11:10:22PM +0800, Rui Zhao wrote: > 2) Only check the tablespace with an absolute path in pg_upgrade. > There are also other solutions, such as supporting the creation of > relative-path tablespace in function CreateTableSpace. But do we > really need relative-path

Re: [feature]COPY FROM enable FORCE_NULL/FORCE_NOT_NULL on all columns

2023-07-31 Thread Zhang Mingli
On Aug 1, 2023, at 03:35, Andrew Dunstan wrote:I was hoping it be able to get to it today but that's not happening. If you want to submit a revised patch as above that will be good. I hope to get to it later this week.HI, Andrew Patch rebased and updated like above, thanks.

Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Michael Paquier
Hi all, 31de7e6 has silenced savepoint names in the query jumbling, and something similar can be done for 2PC transactions once the GID is ignored in TransactionStmt. This leads to the following grouping in pg_stat_statements, for instance, which is something that matters with workloads that

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Tom Lane
Andres Freund writes: > On 2023-07-31 19:11:38 -0400, Tom Lane wrote: >> Huh. Maybe worth reporting as a FreeBSD bug? > Yea. Hoping our local freebsd developer has a suggestion as to which component > to report it to, or even fix it :). You already have a reproducer using just tcl, so I'd

Re: Getting rid of OverrideSearhPath in namespace.c

2023-07-31 Thread Noah Misch
On Mon, Jul 17, 2023 at 05:11:46PM +0300, Aleksander Alekseev wrote: > > As a follow-up for the CVE-2023-2454 fix, I think that it makes sense to > > completely remove unsafe functions > > PushOverrideSearchPath()/PopOverrideSearchPath(), which are not used in the > > core now. > > Please look at

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 19:11:38 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-07-31 18:33:37 -0400, Tom Lane wrote: > >> (And could it be that we had one in the predecessor 13.1 image?) > > > No, I checked, and it's not in there either... It looks like the difference > > is > > that

Re: Avoid possible memory leak (src/common/rmtree.c)

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 08:10:55PM -0300, Ranier Vilela wrote: > Thanks for the commit, Michael. Sorry for the lack of update here. For the sake of the archives, this is f1e9f6b. -- Michael signature.asc Description: PGP signature

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 21:25:06 +, José Neves wrote: > Ok, if I understood you correctly, I start to see where my logic is faulty. > Just to make sure that I got it right, taking the following example again: > > T-1 > INSERT LSN1-1000 > UPDATE LSN2-2000 > UPDATE LSN3-3000 > COMMIT LSN4-4000 > >

Re: Avoid possible memory leak (src/common/rmtree.c)

2023-07-31 Thread Ranier Vilela
Em sex, 28 de jul de 2023 11:54 PM, Michael Paquier escreveu: > On Tue, Jul 25, 2023 at 04:45:22PM +0200, Daniel Gustafsson wrote: > > Skimming the tree there doesn't seem to be any callers which aren't > exiting or > > ereporting on failure so the real-world impact seems low. That being >

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Tom Lane
Andres Freund writes: > On 2023-07-31 18:33:37 -0400, Tom Lane wrote: >> (And could it be that we had one in the predecessor 13.1 image?) > No, I checked, and it's not in there either... It looks like the difference is > that 13.1 reads the UTC zoneinfo in that case, whereas 13.2 doesn't. Huh.

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 18:33:37 -0400, Tom Lane wrote: > Andres Freund writes: > > I saw that CI image builds for freebsd were failing, because 13.1, used > > until > > now, is EOL. Update to 13.2, no problem, I thought. Ran a CI run against > > 13.2 > > - unfortunately that failed. In pltcl of all

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Tom Lane
Andres Freund writes: > I saw that CI image builds for freebsd were failing, because 13.1, used until > now, is EOL. Update to 13.2, no problem, I thought. Ran a CI run against 13.2 > - unfortunately that failed. In pltcl of all places. I tried to replicate this in a freshly-installed 13.2 VM,

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Jeff Davis
On Mon, 2023-07-31 at 13:17 -0400, Joe Conway wrote: > But the analysis of the issue needs to go one step further. Even if > the > search_path does not change from the originally intended one, a newly > created function can shadow the intended one based on argument > coercion > rules. There are

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Jeff Davis
On Mon, 2023-07-31 at 12:53 -0400, Robert Haas wrote: > I agree. I think there are actually two interrelated problems here. > > One is that virtually all code needs to run with the originally > intended search_path rather than some search_path chosen at another > time and maybe by a different

RE: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi Andres, thanks for your reply. Ok, if I understood you correctly, I start to see where my logic is faulty. Just to make sure that I got it right, taking the following example again: T-1 INSERT LSN1-1000 UPDATE LSN2-2000 UPDATE LSN3-3000 COMMIT LSN4-4000 T-2 INSERT LSN1-500 UPDATE LSN2-1500

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Jeff Davis
On Mon, 2023-07-31 at 16:06 -0400, Robert Haas wrote: > if you > include in your search_path a schema to which some other user can > write, you are pretty much agreeing to execute code provided by that > user. Agreed on all counts here. I don't think it's reasonable for us to try to make such a

Re: Correct the documentation for work_mem

2023-07-31 Thread Tristen Raab
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed Hello, I've reviewed and built the documentation for the

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 14:16:22 +, José Neves wrote: > Hi Amit, thanks for the reply. > > In our worker (custom pg replication client), we care only about INSERT, > UPDATE, and DELETE operations, which - sure - may be part of the issue. That seems likely. Postgres streams out changes in commit

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Robert Haas
On Mon, Jul 31, 2023 at 1:18 PM Joe Conway wrote: > But the analysis of the issue needs to go one step further. Even if the > search_path does not change from the originally intended one, a newly > created function can shadow the intended one based on argument coercion > rules. Yeah, this is a

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 12:15:10 -0700, Andres Freund wrote: > It sure looks like freebsd 13.2 tcl is just busted. Notably it can't even > parse what it generates: > > echo 'puts [clock scan [clock format [clock seconds] -format "%Y/%m/%d"] > -format "%Y/%m/%d"]'|tclsh8.6 > > Which works on 13.1

Re: [feature]COPY FROM enable FORCE_NULL/FORCE_NOT_NULL on all columns

2023-07-31 Thread Andrew Dunstan
On 2023-07-26 We 03:03, Zhang Mingli wrote: HI, I've looked at this patch and it looks mostly fine, though I do not intend to commit it myself; perhaps Andrew will. HI, Amit, thanks for review. A few minor things to improve: +  If * is specified, it will be applied in all columns.

pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, I saw that CI image builds for freebsd were failing, because 13.1, used until now, is EOL. Update to 13.2, no problem, I thought. Ran a CI run against 13.2 - unfortunately that failed. In pltcl of all places.

RE: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi Euler, thank you for your reply. Your output is exactly how mine doesn't look like, I don't have such an order - that is - not only under heavy load. Conditions in which this occurs make it challenging to provide detailed information, and will also take a while to trigger. I've sent a

Re: should frontend tools use syncfs() ?

2023-07-31 Thread Nathan Bossart
On Mon, Jul 31, 2023 at 10:51:38AM -0700, Nathan Bossart wrote: > Here is a new version of the patch with documentation updates and a couple > other small improvements. I just realized I forgot to update the --help output for these utilities. I'll do that in the next version of the patch. --

Re: add timing information to pg_upgrade

2023-07-31 Thread Nathan Bossart
On Mon, Jul 31, 2023 at 11:34:57AM +0530, Bharath Rupireddy wrote: > Either of "Checking for \"aclitem\" data type usage" or "Checking for > \"aclitem\" data type in user tables" seems okay to me, however, I > prefer the second wording. Okay. I used the second wording for all the data type

Re: should frontend tools use syncfs() ?

2023-07-31 Thread Nathan Bossart
On Sat, Jul 29, 2023 at 02:40:10PM -0700, Nathan Bossart wrote: > I was about to start a new thread, but I found this one with some good > preliminary discussion. I came to the same conclusion about introducing a > new option instead of using syncfs() by default wherever it is available. > The

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Joe Conway
On 7/31/23 12:53, Robert Haas wrote: On Fri, Jun 30, 2023 at 3:41 AM Jeff Davis wrote: I'm not sure that everyone in this thread realizes just how broken it is to depend on search_path in a functional index at all. And doubly so if it depends on a schema other than pg_catalog in the

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Peter Geoghegan
On Mon, Jul 31, 2023 at 12:24 PM Alena Rybakina wrote: > I noticed that you are going to add CNF->DNF transformation at the index > construction stage. If I understand correctly, you will rewrite > restrictinfo node, > change boolean "AND" expressions to "OR" expressions, but would it be >

Re: Faster "SET search_path"

2023-07-31 Thread Robert Haas
On Sat, Jul 29, 2023 at 11:59 AM Jeff Davis wrote: > Unfortunately, adding a "SET search_path" clause to functions slows > them down. The attached patches close the performance gap > substantially. I haven't reviewed the code but I like the concept a lot. This is badly needed. -- Robert Haas

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Robert Haas
On Fri, Jun 30, 2023 at 3:41 AM Jeff Davis wrote: > I'm not sure that everyone in this thread realizes just how broken it > is to depend on search_path in a functional index at all. And doubly so > if it depends on a schema other than pg_catalog in the search_path. > > Let's also not forget that

Re: POC, WIP: OR-clause support for indexes

2023-07-31 Thread Alena Rybakina
Hi! I think it really helps to speed up the search with similar deep filtering compared to cluster indexes, but do we have cases where we don't use this algorithm because it takes longer than an usual index? I thought about the situation with wide indexes (with a lot of multiple columns) and

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Peter Geoghegan
On Thu, Jul 27, 2023 at 10:00 AM Matthias van de Meent wrote: > My idea is not quite block nested loop join. It's more 'restart the > index scan at the location the previous index scan ended, if > heuristics say there's a good chance that might save us time'. I'd say > it is comparable to the

Re: Request for comment on setting binary format output per session

2023-07-31 Thread Dave Cramer
Dave Cramer On Mon, 10 Jul 2023 at 03:56, Daniel Gustafsson wrote: > > On 25 Apr 2023, at 16:47, Dave Cramer wrote: > > > Patch attached with comments removed > > This patch no longer applies, please submit a rebased version on top of > HEAD. > Rebased see attached > > -- > Daniel

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Alena Rybakina
Hi, all! CNF -> DNF conversion = Like many great papers, the MDAM paper takes one core idea, and finds ways to leverage it to the hilt. Here the core idea is to take predicates in conjunctive normal form (an "AND of ORs"), and convert them into disjunctive normal form (an

Re: proposal: psql: show current user in prompt

2023-07-31 Thread Jelte Fennema
On Mon, 24 Jul 2023 at 21:16, Pavel Stehule wrote: > I don't understand how it can be possible to do it without. I need to > process possible errors, and then I need to read and synchronize protocol. I > didn't inject > this feature to some oher flow, so I need to implement a complete process.

Re: pg_upgrade fails with in-place tablespace

2023-07-31 Thread Junwang Zhao
On Mon, Jul 31, 2023 at 5:36 PM Rui Zhao wrote: > > Hello postgres hackers, > Recently I encountered an issue: pg_upgrade fails when dealing with in-place > tablespace. As we know, pg_upgrade uses pg_dumpall to dump objects and > pg_restore to restore them. The problem seems to be that

Inaccurate comments in ReorderBufferCheckMemoryLimit()

2023-07-31 Thread Masahiko Sawada
Hi all, While reading the code, I realized that the following code comments might not be accurate: /* * Pick the largest transaction (or subtransaction) and evict it from * memory by streaming, if possible. Otherwise, spill to disk. */ if

RE: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi Amit, thanks for the reply. In our worker (custom pg replication client), we care only about INSERT, UPDATE, and DELETE operations, which - sure - may be part of the issue. I can only replicate this with production-level load, not easy to get a real example, but as I'm understanding the

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Euler Taveira
On Sat, Jul 29, 2023, at 8:07 PM, José Neves wrote: > I'm attempting to develop a CDC on top of Postgres, currently using 12, the > last minor, with a custom client, and I'm running into issues with data loss > caused by out-of-order logical replication messages. Can you provide a test case to

Re: Improve join_search_one_level readibilty (one line change)

2023-07-31 Thread Julien Rouhaud
Hi, On Wed, Jun 07, 2023 at 11:02:09AM +0800, 謝東霖 wrote: > Thank you, Julien, for letting me know that cfbot doesn't test txt files. > Much appreciated! Thanks for posting this v2! So unsurprisingly the cfbot is happy with this patch, since it doesn't change the behavior at all. I just have

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 3:06 PM José Neves wrote: > > Hi there, hope to find you well. > > I'm attempting to develop a CDC on top of Postgres, currently using 12, the > last minor, with a custom client, and I'm running into issues with data loss > caused by out-of-order logical replication

Re: Adding a LogicalRepWorker type field

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 3:25 PM Bharath Rupireddy wrote: > > On Mon, Jul 31, 2023 at 7:17 AM Peter Smith wrote: > > > > PROBLEM: > > > > IMO, deducing the worker's type by examining multiple different field > > values seems a dubious way to do it. This maybe was reasonable enough > > when there

Re: [BUG] Fix DETACH with FK pointing to a partitioned table fails

2023-07-31 Thread Alvaro Herrera
On 2023-Jul-05, Jehan-Guillaume de Rorthais wrote: > ALTER TABLE r ATTACH PARTITION r_1 FOR VALUES IN (1); > > The old sub-FKs (below 18289) created in this table to enforce the action > triggers on referenced partitions are not deleted when the table becomes a > partition. Because of this,

Re: New PostgreSQL Contributors

2023-07-31 Thread Matthias van de Meent
On Fri, 28 Jul 2023 at 17:29, Christoph Berg wrote: > > The PostgreSQL contributors team has been looking over the community > activity and, over the first half of this year, has been recognizing > new contributors to be listed on > > https://www.postgresql.org/community/contributors/ > > New

Re: stats test intermittent failure

2023-07-31 Thread Masahiko Sawada
Hi, On Tue, Jul 11, 2023 at 3:35 AM Melanie Plageman wrote: > > Hi, > > Jeff pointed out that one of the pg_stat_io tests has failed a few times > over the past months (here on morepork [1] and more recently here on > francolin [2]). > > Failing test diff for those who prefer not to scroll: > >

Re: Avoid undefined behavior with msvc compiler (src/include/port/pg_bitutils.h)

2023-07-31 Thread John Naylor
On Mon, Jul 31, 2023 at 5:57 PM Tom Lane wrote: > > John Naylor writes: > > Works for me, so done that way for both forward and reverse variants. Since > > the return value is no longer checked in any builds, I thought about > > removing the variable containing it, but it seems best to leave it

Re: logical decoding and replication of sequences, take 2

2023-07-31 Thread Tomas Vondra
On 7/31/23 11:25, Amit Kapila wrote: > On Sat, Jul 29, 2023 at 5:53 PM Tomas Vondra > wrote: >> >> On 7/28/23 14:44, Ashutosh Bapat wrote: >>> On Wed, Jul 26, 2023 at 8:48 PM Tomas Vondra >>> wrote: Anyway, I was thinking about this a bit more, and it seems it's not as

Re: Avoid undefined behavior with msvc compiler (src/include/port/pg_bitutils.h)

2023-07-31 Thread Tom Lane
John Naylor writes: > On Sun, Jul 30, 2023 at 9:45 PM Tom Lane wrote: >> That's basically equivalent to the existing Assert(non_zero). >> I think it'd be okay to drop that one and instead have >> the same Assert condition as other platforms, but having both >> would be redundant. > Works for

Re: Support to define custom wait events for extensions

2023-07-31 Thread Bharath Rupireddy
On Mon, Jul 31, 2023 at 3:54 PM Michael Paquier wrote: > > On Mon, Jul 31, 2023 at 05:10:21PM +0900, Kyotaro Horiguchi wrote: > > +/* > > + * Return the name of an wait event ID for extension. > > + */ > > +static const char * > > +GetWaitEventExtensionIdentifier(uint16 eventId) > > > > This

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 05:10:21PM +0900, Kyotaro Horiguchi wrote: > +/* > + * Return the name of an wait event ID for extension. > + */ > +static const char * > +GetWaitEventExtensionIdentifier(uint16 eventId) > > This looks inconsistent. Shouldn't it be GetWaitEventExtentionName()? This is an

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 01:37:49PM +0530, Bharath Rupireddy wrote: > Do you think it's worth adding a note here in the docs about an > external module defining more than one custom wait event? A pseudo > code if possible or just a note? Also, how about a XXX comment atop > WaitEventExtensionNew

Missing comments/docs about custom scan path

2023-07-31 Thread Etsuro Fujita
Hi, While working on [1], I noticed $SUBJECT: commit e7cb7ee14 failed to update comments for the CustomPath struct in pathnodes.h, and commit f49842d1e failed to update docs about custom scan path callbacks in custom-scan.sgml, IIUC. Attached are patches for updating these, which I created

Re: Adding a LogicalRepWorker type field

2023-07-31 Thread Bharath Rupireddy
On Mon, Jul 31, 2023 at 7:17 AM Peter Smith wrote: > > PROBLEM: > > IMO, deducing the worker's type by examining multiple different field > values seems a dubious way to do it. This maybe was reasonable enough > when there were only 2 types, but as more get added it becomes > increasingly

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-31 Thread John Naylor
On Thu, Jul 27, 2023 at 7:17 AM David Rowley wrote: > > It would be really good if someone with another a newish intel CPU > could test this too. I ran the lotsaints test from last email on an i7-10750H (~3 years old) and got these results (gcc 13.1 , turbo off): REL_15_STABLE: latency average

回复:pg_rewind fails with in-place tablespace

2023-07-31 Thread Rui Zhao
Sorry for the delay in responding to this matter as I have been waiting for another similar subject to approved by a moderator. Upon review, I am satisfied with the proposed solution and believe that checking absolute path is better than hard coding with "pg_tblspc/". I think we have

CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi there, hope to find you well. I'm attempting to develop a CDC on top of Postgres, currently using 12, the last minor, with a custom client, and I'm running into issues with data loss caused by out-of-order logical replication messages. The problem is as follows: postgres streams A, B, D, G,

pg_upgrade fails with in-place tablespace

2023-07-31 Thread Rui Zhao
Hello postgres hackers, Recently I encountered an issue: pg_upgrade fails when dealing with in-place tablespace. As we know, pg_upgrade uses pg_dumpall to dump objects and pg_restore to restore them. The problem seems to be that pg_dumpall is dumping in-place tablespace as relative path, which

Re: logical decoding and replication of sequences, take 2

2023-07-31 Thread Amit Kapila
On Sat, Jul 29, 2023 at 5:53 PM Tomas Vondra wrote: > > On 7/28/23 14:44, Ashutosh Bapat wrote: > > On Wed, Jul 26, 2023 at 8:48 PM Tomas Vondra > > wrote: > >> > >> Anyway, I was thinking about this a bit more, and it seems it's not as > >> difficult to use the page LSN to ensure sequences

Re: postgres_fdw: wrong results with self join + enable_nestloop off

2023-07-31 Thread Richard Guo
On Fri, Jul 28, 2023 at 4:56 PM Etsuro Fujita wrote: > Cool! I pushed the first patch after polishing it a little bit, so > here is a rebased version of the second patch, in which I modified the > ForeignPath and CustomPath cases in reparameterize_path_by_child() to > reflect the new members

Re: Support to define custom wait events for extensions

2023-07-31 Thread Kyotaro Horiguchi
At Mon, 31 Jul 2023 16:28:16 +0900, Michael Paquier wrote in > Attaching a v11 based on Bharath's feedback and yours, for now. I > have also applied the addition of the two masking variables in > wait_event.c separately with 7395a90. +/* + * Return the name of an wait event ID for extension.

Re: Support to define custom wait events for extensions

2023-07-31 Thread Bharath Rupireddy
On Mon, Jul 31, 2023 at 12:58 PM Michael Paquier wrote: > > > Attaching a v11 based on Bharath's feedback and yours, for now. I > have also applied the addition of the two masking variables in > wait_event.c separately with 7395a90. +uint32 WaitEventExtensionNew(void) + + Next, each process

Re: Avoid undefined behavior with msvc compiler (src/include/port/pg_bitutils.h)

2023-07-31 Thread John Naylor
On Sun, Jul 30, 2023 at 9:45 PM Tom Lane wrote: > > John Naylor writes: > > It seems that we should have "Assert(word != 0);" at the top, which matches > > the other platforms anyway, so I'll add that. > > That's basically equivalent to the existing Assert(non_zero). > I think it'd be okay to

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
On 2023-07-31 16:28, Michael Paquier wrote: On Mon, Jul 31, 2023 at 03:53:27PM +0900, Masahiro Ikeda wrote: /* This should only be called for user-defined wait event. */ if (eventId < NUM_BUILTIN_WAIT_EVENT_EXTENSION) ereport(ERROR,

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 03:53:27PM +0900, Masahiro Ikeda wrote: > I think the order in which they are mentioned should be matched. I mean that > - so an LWLock or Extension wait > + so an Extension or LWLock wait Makes sense. > /* This should only be called for user-defined wait

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 12:07:40PM +0530, Bharath Rupireddy wrote: > We're not giving up and returning an unknown state in the v10 patch > unlike v9, no? This comment needs to change. Right. Better to be consistent with lwlock.c here. >> No, it cannot because we need the custom wait event

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
On 2023-07-31 10:10, Michael Paquier wrote: Attached is a new version. Thanks for all the improvements. I have some comments for v10. (1) - Extensions can add LWLock types to the list shown in - . In some cases, the name + Extensions can add Extension and +

Re: Support to define custom wait events for extensions

2023-07-31 Thread Bharath Rupireddy
On Mon, Jul 31, 2023 at 6:40 AM Michael Paquier wrote: > > You are right that I am making things inconsistent here. Having a > behavior close to the existing LWLock and use "extension" when the > event cannot be found makes the most sense. I have been a bit > confused with the wording though of

Re: add timing information to pg_upgrade

2023-07-31 Thread Bharath Rupireddy
On Sun, Jul 30, 2023 at 2:44 AM Nathan Bossart wrote: > > On Sat, Jul 29, 2023 at 12:17:40PM +0530, Bharath Rupireddy wrote: > > While on this, I noticed a thing unrelated to your patch that there's > > no space between the longest status message of size 60 bytes and ok - > > 'Checking for