Re: Slow standby snapshot

2021-08-02 Thread Andrey Borodin
Hi, > 3 авг. 2021 г., в 03:01, Andres Freund написал(а): > > Hi, > > On 2021-08-03 00:07:23 +0300, Michail Nikolaev wrote: >> The main idea is simple optimistic optimization - store offset to next >> valid entry. So, in most cases, we could just skip all the gaps. >> Of course, it adds some

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread vignesh C
On Mon, Aug 2, 2021 at 1:13 PM Masahiko Sawada wrote: > > On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com > wrote: > > > > On Thursday, July 29, 2021 10:50 AM Masahiko Sawada > > wrote: > > > On Thu, Jul 8, 2021 at 3:55 PM osumi.takami...@fujitsu.com > > > wrote: > > > > When the

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 11:47 AM Amit Kapila wrote: > > On Mon, Aug 2, 2021 at 1:13 PM Masahiko Sawada wrote: > > > > On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > > > > Accordingly, I'm thinking to have unsuccessful and successful stats on > > > the sub

Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-02 Thread Soumyadeep Chakraborty
Hello, I came across this issue while I was tweaking a TAP test with Ashwin for this thread [1]. We noticed that while the startup process waits for a recovery delay, it does not respect changes to the recovery_min_apply_delay GUC. So: // Standby is waiting on recoveryWakeupLatch with a large

Sort keys are omitted incorrectly after grouping sets

2021-08-02 Thread Richard Guo
Hi hackers, In each PathKey we have an EC representing the value being sorted on. This works correctly and efficiently in scan/join planning since the members in an EC are always equal to each other. However, after applying GROUP BY grouping sets, the members in an EC may not be that equivalent

Remove unused 'len' from pg_stat_recv_* functions

2021-08-02 Thread Masahiko Sawada
Hi all, While working on a patch adding new stats, houzj pointed out that 'len' function arguments of all pgstat_recv_* functions are not used at all. Looking at git history, pgstat_recv_* functions have been having ‘len’ since the stats collector was introduced by commit 140ddb78fe 20 years ago

Re: row filtering for logical replication

2021-08-02 Thread Peter Smith
FYI - v20 --> v21 (Only very minimal changes) * I noticed that the v20 TAP test (023_row_filter.pl) began failing due to a recent commit [1], so I have rebased it to keep the cfbot happy. -- [1] https://github.com/postgres/postgres/commit/201a76183e2056c2217129e12d68c25ec9c559c8 Kind

RE: [PATCH]Comment improvement in publication.sql

2021-08-02 Thread tanghy.f...@fujitsu.com
On Monday, August 2, 2021 11:56 PM vignesh C wrote: > > Few minor suggestions: > 1) Should we change below to "fail - tables can't be added, dropped or > set to "FOR ALL TABLES" publications" > --- fail - can't add to for all tables publication > +-- fail - tables can't be added to or dropped

Re: A micro-optimisation for ProcSendSignal()

2021-08-02 Thread Andres Freund
Hi, On 2021-08-03 13:44:58 +1200, Thomas Munro wrote: > New idea. Instead of adding pgprocno to SERIALIZABLEXACT (which we > should really be trying to shrink, not grow), let's look it up by > vxid->backendId. I didn't consider that before, because I was trying > not to get tangled up with

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread Amit Kapila
On Mon, Aug 2, 2021 at 1:13 PM Masahiko Sawada wrote: > > On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com > wrote: > > > > > > Accordingly, I'm thinking to have unsuccessful and successful stats on the > > sub side. > > Sawada-san is now implementing a new view in [1]. > > Do you

Re: A micro-optimisation for ProcSendSignal()

2021-08-02 Thread Soumyadeep Chakraborty
Hey Thomas, On Mon, Aug 2, 2021 at 6:45 PM Thomas Munro wrote: > > Hi Soumyadeep, > > On Sat, Jul 24, 2021 at 5:26 PM Soumyadeep Chakraborty > wrote: > > On Tue, Jul 20, 2021 at 10:40 PM Thomas Munro > > wrote: > > > I wonder why we need this member anyway, when you can compute it from > > >

Re: archive status ".ready" files may be created too early

2021-08-02 Thread Kyotaro Horiguchi
At Mon, 2 Aug 2021 23:28:19 +, "Bossart, Nathan" wrote in > On 8/2/21, 2:42 PM, "Alvaro Herrera" wrote: > > I think it's okay to make lastNotifiedSeg protected by just info_lck, > > and RecordBoundaryMap protected by just ArchNotifyLock. It's okay to > > acquire the spinlock inside the

Dividing line between before_shmem_exit and on_shmem_exit?

2021-08-02 Thread Andres Freund
Hi, I don't think I really understand what decides what should use before_shmem_exit() and what into on_shmem_exit(). The existing comments say: /* * before_shmem_exit * * Register early callback to

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Thomas Munro
On Tue, Aug 3, 2021 at 9:52 AM Thomas Munro wrote: > On Tue, Aug 3, 2021 at 1:17 AM Robert Haas wrote: > > That's great. I just realized that this leaves us with identical > > RequestCheckpoint() calls in two nearby places. Is there any reason > > not to further simplify as in the attached? > >

Re: 2021-07 CF now in progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 10:50 AM Masahiko Sawada wrote: > > On Tue, Aug 3, 2021 at 10:43 AM Tom Lane wrote: > > > > Masahiko Sawada writes: > > > I can close this commitfest. But should we mark uncommitted patches as > > > "Moved to next CF" or "Returned with Feedback" beforehand? > > > > Should

fix DECLARE tab completion

2021-08-02 Thread Shinya11.Kato
Hi! In the discussion of [1], the option ASENSITIVE was added to DECLARE. I have improved its tab completion and attached a patch. Do you think? [1] https://www.postgresql.org/message-id/flat/96ee8b30-9889-9e1b-b053-90e10c050e85%40enterprisedb.com Regards, Shinya Kato

Re: 2021-07 CF now in progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 10:43 AM Tom Lane wrote: > > Masahiko Sawada writes: > > I can close this commitfest. But should we mark uncommitted patches as > > "Moved to next CF" or "Returned with Feedback" beforehand? > > Should be moved to next CF, or at least most of them should be. > (But I doubt

Re: A micro-optimisation for ProcSendSignal()

2021-08-02 Thread Thomas Munro
Hi Soumyadeep, On Sat, Jul 24, 2021 at 5:26 PM Soumyadeep Chakraborty wrote: > On Tue, Jul 20, 2021 at 10:40 PM Thomas Munro wrote: > > I wonder why we need this member anyway, when you can compute it from > > the address... #define GetPGProcNumber(p) ((p) - ProcGlobal->allProcs) > > or

Re: 2021-07 CF now in progress

2021-08-02 Thread Tom Lane
Masahiko Sawada writes: > I can close this commitfest. But should we mark uncommitted patches as > "Moved to next CF" or "Returned with Feedback" beforehand? Should be moved to next CF, or at least most of them should be. (But I doubt anyone wants to try to kill off patches that aren't going

Re: 2021-07 CF now in progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 12:34 AM Ibrar Ahmed wrote: > > > > On Mon, Jul 26, 2021 at 5:52 PM Ibrar Ahmed wrote: >> >> >> >> On Mon, Jul 19, 2021 at 4:37 PM Ibrar Ahmed wrote: >>> >>> >>> >>> On Mon, Jul 12, 2021 at 4:59 PM Ibrar Ahmed wrote: Hackers, The Commitfest

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Mon, May 10, 2021 at 1:31 PM vignesh C wrote: > ... > > 2) I felt we can change lsn data type from Int64 to XLogRecPtr > + > +Int64 > + > +The LSN of the prepare. > + > + > + > + > +Int64 > + > +The end LSN of the transaction. > + > + > > 3) I felt we can change

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
Please find attached the latest patch set v102* Differences: * Rebased to HEAD @ today. * This is a documentation change only. A recent commit [1] has changed the documentation style for the message formats slightly to annotate the data types. For consistency, the same style change needs to be

Re: Record a Bitmapset of non-pruned partitions

2021-08-02 Thread David Rowley
On Mon, 2 Aug 2021 at 20:16, David Rowley wrote: > If anyone has any objections to both the v4 0001 and 0002 patch, can > they let me know soon. I'm currently seeing no reason that they can't > go in. I've now pushed both of these. Thanks for the reviews. David

RE: make MaxBackends available in _PG_init

2021-08-02 Thread wangsh.f...@fujitsu.com
Hi, Bossart, Nathan wrote: > I just encountered the same thing, so I am bumping this thread. Thank you for bumping this thread. > > I've wondered, independent of this thread, about not making MaxBackends > > externally visible, and requiring a function call to access it. It should be > >

Re: archive status ".ready" files may be created too early

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 2:42 PM, "Alvaro Herrera" wrote: > I think it's okay to make lastNotifiedSeg protected by just info_lck, > and RecordBoundaryMap protected by just ArchNotifyLock. It's okay to > acquire the spinlock inside the lwlock-protected area, as long as we > make sure never to do the opposite.

Re: disfavoring unparameterized nested loops

2021-08-02 Thread Mike Klaas
I think that it is worth paying more than nothing to avoid plans that are so far from optimal that they might as well take forever to execute I recently came across this article from 2016 that expounded upon a bad plan of the sort discussed in this thread:

Re: Use generation context to speed up tuplesorts

2021-08-02 Thread David Rowley
On Sat, 31 Jul 2021 at 08:38, Andres Freund wrote: > There is at least one path using tuplecontext that reaches code that > could end up freeing memory to a significant enough degree to care about > generation.c effectively not using that memory: >

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 3:42 PM, "Andres Freund" wrote: > I've wondered, independent of this thread, about not making MaxBackends > externally visible, and requiring a function call to access it. It should be > easier to find cases of misuse if we errored out when accessed at the wrong > time. And we could

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 22:35:13 +, Bossart, Nathan wrote: > On 8/2/21, 3:12 PM, "Andres Freund" wrote: > > I think this is overblown. We already size resources *after* > > shared_preload_libraries' _PG_init() runs, because that's the whole point of > > shared_preload_libraries. What's proposed in

Re: Make relfile tombstone files conditional on WAL level

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 16:03:31 -0400, Robert Haas wrote: > The two most principled solutions to this problem that I can see are > (1) remove wal_level=minimal and I'm personally not opposed to this. It's not practically relevant and makes a lot of stuff more complicated. We imo should rather focus

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 3:12 PM, "Andres Freund" wrote: > I think this is overblown. We already size resources *after* > shared_preload_libraries' _PG_init() runs, because that's the whole point of > shared_preload_libraries. What's proposed in this thread is to *disallow* > increasing resource usage in s_p_l

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-08-02 Thread Melanie Plageman
On Fri, Jun 4, 2021 at 5:52 PM Alvaro Herrera wrote: > > On 2021-Apr-12, Melanie Plageman wrote: > > > As for the way I have recorded strategy writes -- it is quite inelegant, > > but, I wanted to make sure that I only counted a strategy write as one > > in which the backend wrote out the dirty

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 21:57:18 +, Bossart, Nathan wrote: > On 8/2/21, 1:37 PM, "Andres Freund" wrote: > > On 2021-08-02 16:06:15 -0400, Robert Haas wrote: > >> I think this is a good idea. Anyone object? > > > > I'm not so sure it's a good idea. I've seen several shared_preload_library > > using

Re: Slow standby snapshot

2021-08-02 Thread Andres Freund
Hi, On 2021-08-03 00:07:23 +0300, Michail Nikolaev wrote: > The main idea is simple optimistic optimization - store offset to next > valid entry. So, in most cases, we could just skip all the gaps. > Of course, it adds some additional impact for workloads without long > (few seconds) transactions

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 1:37 PM, "Andres Freund" wrote: > On 2021-08-02 16:06:15 -0400, Robert Haas wrote: >> I think this is a good idea. Anyone object? > > I'm not so sure it's a good idea. I've seen several shared_preload_library > using extensions that adjust some GUCs (e.g. max_prepared_transactions) >

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Thomas Munro
On Tue, Aug 3, 2021 at 1:17 AM Robert Haas wrote: > That's great. I just realized that this leaves us with identical > RequestCheckpoint() calls in two nearby places. Is there any reason > not to further simplify as in the attached? LGTM.

Re: archive status ".ready" files may be created too early

2021-08-02 Thread Alvaro Herrera
On 2021-Jul-31, Bossart, Nathan wrote: > This is why I was trying to get away with just using info_lck for > reading lastNotifiedSeg. ArchNotifyLock is mostly intended to protect > RecordBoundaryMap. However, since lastNotifiedSeg is used in > functions like GetLatestRecordBoundarySegment()

Re: Implementing Incremental View Maintenance

2021-08-02 Thread Zhihong Yu
On Sun, Aug 1, 2021 at 11:30 PM Yugo NAGATA wrote: > Hi hackers, > > On Mon, 19 Jul 2021 09:24:30 +0900 > Yugo NAGATA wrote: > > > On Wed, 14 Jul 2021 21:22:37 +0530 > > vignesh C wrote: > > > > The patch does not apply on Head anymore, could you rebase and post a > > > patch. I'm changing the

Re: [PATCH] proposal for regexp_count, regexp_instr, regexp_substr and regexp_replace

2021-08-02 Thread Gilles Darold
Le 02/08/2021 à 01:21, Tom Lane a écrit : > Gilles Darold writes: >> [ v5-0001-regexp-foo-functions.patch ] > I've gone through this whole patch now, and found quite a lot that I did > not like. In no particular order: > > * Wrapping parentheses around the user's regexp doesn't work. It can >

Re: Slow standby snapshot

2021-08-02 Thread Michail Nikolaev
Hello. > I have tried such an approach but looks like it is not effective, > probably because of CPU caching issues. It was a mistake by me. I have repeated the approach and got good results with small and a non-invasive patch. The main idea is simple optimistic optimization - store offset to

Release 13 of the PostgreSQL BuildFarm client

2021-08-02 Thread Andrew Dunstan
I have pushed Release 13 of the PostgreSQL BuildFarm client. Change highlights: * add tests for a few cases that were previously missing * more fine-grained control over which TAP test sets run * --no-keepall can be specified on the command line * repair a repo if it crashed during a

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 16:06:15 -0400, Robert Haas wrote: > On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan wrote: > > I just encountered the same thing, so I am bumping this thread. I was > > trying to use MaxBackends in a call to RequestAddinShmemSpace() in a > > _PG_init() function for a module,

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Robert Haas wrote: > On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan wrote: > > > > I think the comments about needing to register background workers > > before initializing MaxBackends have been incorrect since the addition > > of max_worker_processes in v9.4 (6bc8ef0b). > I

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan wrote: > I just encountered the same thing, so I am bumping this thread. I was > trying to use MaxBackends in a call to RequestAddinShmemSpace() in a > _PG_init() function for a module, but since MaxBackends is not yet > initialized, you essentially

Re: Make relfile tombstone files conditional on WAL level

2021-08-02 Thread Robert Haas
On Thu, Jun 10, 2021 at 6:47 AM Heikki Linnakangas wrote: > It would indeed be nice to have some other mechanism to prevent the > issue with wal_level=minimal, the tombstone files feel hacky and > complicated. Maybe a new shared memory hash table to track the > relfilenodes of dropped tables.

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Andres Freund wrote: > On Mon, Aug 2, 2021, at 12:12, Alvaro Herrera wrote: > > Hmm, I don't remember that an shmem-unconnected bgworker first connected > > to it and then let go. It seems weird to do it that way. My intention, > > as far as I recall, is that they would just

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On Mon, Aug 2, 2021, at 12:12, Alvaro Herrera wrote: > On 2021-Aug-02, Andres Freund wrote: > > I do think there's some potential gains in simplicity and robustness > > that are made mildly harder by a subprocess that first attaches and > > detaches from shm (it's the only case where we can't

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Andres Freund wrote: > > When I included this case I was thinking in tasks which would just run > > stuff not directly connected to data. Something like a sub-daemon: say > > a connection pooler, which is a bgworker just so that it starts and > > stops together with postmaster,

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On Mon, Aug 2, 2021, at 11:08, Alvaro Herrera wrote: > On 2021-Aug-02, Tom Lane wrote: > > > Robert Haas writes: > > > If you're saying that this code has been 100% broken for 7 years and > > > nobody's noticed until now, then that suggests that nobody actually > > > uses

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Tom Lane wrote: > Robert Haas writes: > > If you're saying that this code has been 100% broken for 7 years and > > nobody's noticed until now, then that suggests that nobody actually > > uses non-shmem-connected bgworkers. I sort of hate to give up on that > > concept but if

Re: straightening out backend process startup

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 12:57:36 -0400, Tom Lane wrote: > Andres Freund writes: > > I think there's quite a few different issues around this - here I'm just > > trying to tackle a few of the most glaring (to me): > > No opinion yet about most of this, but I did want to react to > > > In fact, I

Re: straightening out backend process startup

2021-08-02 Thread Tom Lane
Andres Freund writes: > I think there's quite a few different issues around this - here I'm just > trying to tackle a few of the most glaring (to me): No opinion yet about most of this, but I did want to react to > In fact, I think there's a good argument to be made that we should > entirely

Passing data out of indexam or tableam

2021-08-02 Thread Chris Cleveland
Hi, I'm attempting to develop a new type of full-text index. I've started building a new index access method. Is it possible to calculate values during the index scan and then make them available for display in a select clause? For example, SELECT title, score, total_hits FROM mytable WHERE

straightening out backend process startup

2021-08-02 Thread Andres Freund
Hi, I've previously complained ([1]) that process initialization has gotten very complicated. I hit this once more last week when trying to commit one of the shared memory stats pieces... I think there's quite a few different issues around this - here I'm just trying to tackle a few of the most

Re: Minimal logical decoding on standbys

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 16:45:23 +0200, Drouvot, Bertrand wrote: > On 7/27/21 7:22 PM, Andres Freund wrote: > > On 2021-07-27 09:23:48 +0200, Drouvot, Bertrand wrote: > > > Thanks for the warning, rebase done and new v21 version attached. > > Did you have a go at fixing the walsender race conditions I

Re: [PATCH]Comment improvement in publication.sql

2021-08-02 Thread vignesh C
On Mon, Aug 2, 2021 at 3:31 PM tanghy.f...@fujitsu.com wrote: > > Hi Hackers > > When review and test another patch at [1], I found some comments in existing > test code of " src/test/regress/sql/publication.sql " is a little bit > confused. > Attached a patch to fix them, please take a check.

Re: Corrected documentation of data type for the logical replication message formats.

2021-08-02 Thread vignesh C
On Mon, Aug 2, 2021 at 9:10 PM Tom Lane wrote: > > Peter Smith writes: > > I agree. The specified value looks better when it comes first, as you did > > it. > > Actually, it looks to me like we don't have to resolve the question of > which should come first, because I don't see any cases where

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 11:00:49 -0400, Tom Lane wrote: > Robert Haas writes: > > If you're saying that this code has been 100% broken for 7 years and > > nobody's noticed until now, then that suggests that nobody actually > > uses non-shmem-connected bgworkers. I sort of hate to give up on that > >

Re: Minimal logical decoding on standbys

2021-08-02 Thread Ronan Dunklau
Le lundi 2 août 2021, 17:31:46 CEST Drouvot, Bertrand a écrit : > > I think the beahviour when dropping a database on the primary should be > > documented, and proper procedures for handling it correctly should be > > suggested. > > > > Something along the lines of: > > > > "If a database is

Re: Corrected documentation of data type for the logical replication message formats.

2021-08-02 Thread Tom Lane
Peter Smith writes: > I agree. The specified value looks better when it comes first, as you did it. Actually, it looks to me like we don't have to resolve the question of which should come first, because I don't see any cases where it's useful to have both. I don't agree with appending "uint8"

Re: 2021-07 CF now in progress

2021-08-02 Thread Ibrar Ahmed
On Mon, Jul 26, 2021 at 5:52 PM Ibrar Ahmed wrote: > > > On Mon, Jul 19, 2021 at 4:37 PM Ibrar Ahmed wrote: > >> >> >> On Mon, Jul 12, 2021 at 4:59 PM Ibrar Ahmed >> wrote: >> >>> >>> Hackers, >>> >>> The Commitfest 2021-07 is now in progress. It is one of the biggest one. >>> Total number of

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Tom Lane
Robert Haas writes: > If you're saying that this code has been 100% broken for 7 years and > nobody's noticed until now, then that suggests that nobody actually > uses non-shmem-connected bgworkers. I sort of hate to give up on that > concept but if we've really gone that many years without

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 2:51 AM Andres Freund wrote: > which presents a problem: We've initialized all kind of references to shared > memory, own a PGPROC, but have detached from shared memory. > > In practice this will lead pretty quickly to a segfault, because process exit > will run proc_exit

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 9:51 AM Jakub Wartak wrote: > BTW, if you now there's this big push for refactoring StartupXLOG() then what > frustrating^H^H^H^H^H could be done better - at least from end-user point of > view - > is that there is lack of near real time cyclic messages (every 1min?)

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 9:40 AM Amul Sul wrote: > > That's great. I just realized that this leaves us with identical > > RequestCheckpoint() calls in two nearby places. Is there any reason > > not to further simplify as in the attached? > > > +1, also, can we just get rid off "promoted" flag? The

[BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION

2021-08-02 Thread houzj.f...@fujitsu.com
Hi hackers, When testing some other logical replication related patches, I found two unexpected behaviours about ALTER SUBSCRIPTION ADD/DROP PUBLICATION. (1) when I execute the following sqls[1], the data of tables that registered to 'pub' wasn't copied to the subscriber side which means

RE: Background writer and checkpointer in crash recovery

2021-08-02 Thread Jakub Wartak
> On Fri, Jul 30, 2021 at 4:00 PM Andres Freund wrote: > > I don't agree with that? If (user+system) << wall then it is very > > likely that recovery is IO bound. If system is a large percentage of > > wall, then shared buffers is likely too small (or we're replacing the > > wrong > > buffers)

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Amul Sul
On Mon, Aug 2, 2021 at 6:47 PM Robert Haas wrote: > > On Mon, Aug 2, 2021 at 1:37 AM Thomas Munro wrote: > > I pushed 0001. > > That's great. I just realized that this leaves us with identical > RequestCheckpoint() calls in two nearby places. Is there any reason > not to further simplify as in

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 1:37 AM Thomas Munro wrote: > I pushed 0001. That's great. I just realized that this leaves us with identical RequestCheckpoint() calls in two nearby places. Is there any reason not to further simplify as in the attached? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Fri, Jul 30, 2021 at 4:00 PM Andres Freund wrote: > I don't agree with that? If (user+system) << wall then it is very likely > that recovery is IO bound. If system is a large percentage of wall, then > shared buffers is likely too small (or we're replacing the wrong > buffers) because you

Re: .ready and .done files considered harmful

2021-08-02 Thread Dipesh Pandit
Hi, > I think what you are saying is true before v14, but not in v14 and master. Yes, we can use archiver specific shared memory. Thanks. > I don't think it's great that we're using up SIGINT for this purpose. > There aren't that many signals available at the O/S level that we can > use for our

Update of partition key on foreign server

2021-08-02 Thread Илья Гладышев
Hi, I am currently looking into a partition constraint violation that occurs on update of a partition key on a foreign server. To reproduce it you can run: On server 1 using port 5432: create extension postgres_fdw; create table players (id integer, name text) partition by list(name); create

Re: Minimal logical decoding on standbys

2021-08-02 Thread Ronan Dunklau
Le mardi 27 juillet 2021, 09:23:48 CEST Drouvot, Bertrand a écrit : > Thanks for the warning, rebase done and new v21 version attached. > > Bertrand Hello, I've taken a look at this patch, and it looks like you adressed every prior remark, including the race condition Andres was worried about.

Re: Use generation context to speed up tuplesorts

2021-08-02 Thread David Rowley
On Sat, 31 Jul 2021 at 14:34, Tomas Vondra wrote: > I spent a bit of time hacking on the Generation context, adding the two > improvements discussed in this thread: > > 1) internal handling of block sizes, similar to what AllocSet does (it > pretty much just copies parts of it) > > 2) keeper

Re: Speeding up GIST index creation for tsvectors

2021-08-02 Thread John Naylor
On Sun, Aug 1, 2021 at 11:41 PM Amit Khandekar wrote: > > > FWIW, I anticipate some push back from the community because of the fact that the optimization relies on statistical phenomena. > > I dug into this issue for tsvector type. Found out that it's the way > in which the sign array elements

[PATCH]Comment improvement in publication.sql

2021-08-02 Thread tanghy.f...@fujitsu.com
Hi Hackers When review and test another patch at [1], I found some comments in existing test code of " src/test/regress/sql/publication.sql " is a little bit confused. Attached a patch to fix them, please take a check. Here is the detail: Existing code: CREATE TABLE testpub_tbl2 (id serial

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Amit Kapila
On Sun, Aug 1, 2021 at 3:51 PM Peter Smith wrote: > > On Sun, Aug 1, 2021 at 3:05 AM vignesh C wrote: > > > > On Sat, Jul 31, 2021 at 11:12 AM Ajin Cherian wrote: > > > > > > On Sat, Jul 31, 2021 at 2:39 PM Amit Kapila > > > wrote: > > > > > > > Here, the test is expecting 2 prepared

Re: logical replication empty transactions

2021-08-02 Thread Amit Kapila
On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian wrote: > Let's first split the patch for prepared and non-prepared cases as that will help to focus on each of them separately. BTW, why haven't you considered implementing point 1b as explained by Andres in his email [1]? I think we can send a

Re: Record a Bitmapset of non-pruned partitions

2021-08-02 Thread David Rowley
On Mon, 2 Aug 2021 at 00:31, David Rowley wrote: > I had another self review of these and I'm pretty happy with them. I'm > quite glad to see the performance of querying a single partition of a > table with large numbers of partitions no longer tails off as much as > it used to. I did some

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 4:33 PM Greg Nancarrow wrote: > > On Fri, Jul 30, 2021 at 2:02 PM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > A few minor comments: > > (1) doc/src/sgml/protocol.sgml > > In the following description,

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 3:18 PM vignesh C wrote: > > On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > Differences: > > > > * Rebased to HEAD @ today (needed because some recent commits [1][2] broke

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread Masahiko Sawada
On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com wrote: > > On Thursday, July 29, 2021 10:50 AM Masahiko Sawada > wrote: > > On Thu, Jul 8, 2021 at 3:55 PM osumi.takami...@fujitsu.com > > wrote: > > > When the current HEAD fails during logical decoding, the failure > > > increments

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 6:25 PM tanghy.f...@fujitsu.com wrote: > > On Friday, July 30, 2021 12:02 PM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > Thanks for your patch. A few comments on the test file: > > 1.

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Sat, Jul 31, 2021 at 9:36 PM Amit Kapila wrote: > > On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > Few minor comments: > 1. > CREATE SUBSCRIPTION regress_testsub CONNECTION > 'dbname=regress_doesnotexist' PUBLICATION testpub

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
Please find attached the latest patch v101 Differences: * Rebased to HEAD @ today. * Addresses all v100 review comments from Vignesh [1], Greg [2], Tang [3], and Amit [2]. [1] https://www.postgresql.org/message-id/CALDaNm2N3qgSv3XyHW%2Bop_SJcLmz1s%3D0jJc-taxUmeEBXW5EPw%40mail.gmail.com

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On 2021-08-01 23:51:16 -0700, Andres Freund wrote: > In practice this will lead pretty quickly to a segfault, because process exit > will run proc_exit callbacks, which in turn will try to do a ProcKill(). Or > logging dereferences MyProc, or ... > > It seems the above code block would need

EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, While looking at a patch I noticed that SubPostmasterMain() for bgworkers unconditionally does /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */ InitProcess(); which presents a problem, because StartBackgroundWorker() then does /* *

RE: Parallel Inserts (WAS: [bug?] Missed parallel safety checks..)

2021-08-02 Thread houzj.f...@fujitsu.com
On August 2, 2021 2:04 PM Greg Nancarrow wrote: > On Mon, Aug 2, 2021 at 2:52 PM Amit Kapila wrote: > > > > On Fri, Jul 30, 2021 at 6:53 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Friday, July 30, 2021 2:52 PM Greg Nancarrow > wrote: > > > > On Fri, Jul 30, 2021 at 4:02 PM Amit Kapila

Re: Skip partition tuple routing with constant partition key

2021-08-02 Thread Amit Langote
I noticed that there is no CF entry for this, so created one in the next CF: https://commitfest.postgresql.org/34/3270/ Rebased patch attached. v8-0001-Teach-get_partition_for_tuple-to-cache-bound-offs.patch Description: Binary data

Re: Parallel Inserts (WAS: [bug?] Missed parallel safety checks..)

2021-08-02 Thread Greg Nancarrow
On Mon, Aug 2, 2021 at 2:52 PM Amit Kapila wrote: > > On Fri, Jul 30, 2021 at 6:53 PM houzj.f...@fujitsu.com > wrote: > > > > On Friday, July 30, 2021 2:52 PM Greg Nancarrow wrote: > > > On Fri, Jul 30, 2021 at 4:02 PM Amit Kapila > > > wrote: > > > > > > > > > Besides, I think we need a new