Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-08-09 Thread Michael Paquier
On Thu, Aug 10, 2023 at 02:47:51PM +0900, Kyotaro Horiguchi wrote: > At Thu, 10 Aug 2023 13:33:48 +0900, Michael Paquier > wrote in >> On Thu, Aug 10, 2023 at 10:15:40AM +0900, Kyotaro Horiguchi wrote: >> > At Thu, 10 Aug 2023 10:00:17 +0900 (JST), Kyotaro Horiguchi >> > wrote in >> >> We hav

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-08-09 Thread Kyotaro Horiguchi
At Thu, 10 Aug 2023 13:33:48 +0900, Michael Paquier wrote in > On Thu, Aug 10, 2023 at 10:15:40AM +0900, Kyotaro Horiguchi wrote: > > At Thu, 10 Aug 2023 10:00:17 +0900 (JST), Kyotaro Horiguchi > > wrote in > >> We have treated every kind of broken data as end-of-recovery, like > >> incorrect

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-08-09 Thread Peter Smith
Hi Melih, FYI -- The same testing was repeated but this time PG was configured to say synchronous_commit=on. Other factors and scripts were the same as before --- busy apply, 5 runs, 4 workers, 1000 inserts/tx, 100 empty tables, etc. There are still more xlog records seen for the v26 patch, but n

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-08-09 Thread Amit Kapila
On Mon, Aug 7, 2023 at 3:46 PM Amit Kapila wrote: > > On Mon, Aug 7, 2023 at 1:06 PM Julien Rouhaud wrote: > > > > On Mon, Aug 07, 2023 at 12:42:33PM +0530, Amit Kapila wrote: > > > On Mon, Aug 7, 2023 at 11:29 AM Julien Rouhaud wrote: > > > > > > > > Unless I'm missing something I don't see wha

Re: Fix pg_stat_reset_single_table_counters function

2023-08-09 Thread Masahiro Ikeda
On 2023-08-01 15:23, Masahiro Ikeda wrote: Hi, My colleague, Mitsuru Hinata (in CC), found the following issue. The documentation of pg_stat_reset_single_table_counters() says pg_stat_reset_single_table_counters ( oid ) → void Resets statistics for a single table or index in the current databa

Re: [PATCH] Add loongarch native checksum implementation.

2023-08-09 Thread John Naylor
On Tue, Aug 8, 2023 at 2:22 PM YANG Xudong wrote: > > On 2023/8/8 14:38, John Naylor wrote: > > > v4 0001 is the same as v3, but with a draft commit message. I will > > squash and commit this week, unless there is additional feedback. > > Looks good to me. Thanks for the additional patch. I pushe

Re: PG 16 draft release notes ready

2023-08-09 Thread Pavel Luzanov
On 09.08.2023 21:06, Bruce Momjian wrote: On Sun, Jul 23, 2023 at 02:09:17PM +0300, Pavel Luzanov wrote: Please consider to add item to the psql section: Add psql \drg command to display role grants and remove the "Member of" column from \du & \dg altogether (d65ddaca) The release notes are on

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-08-09 Thread Amit Kapila
On Wed, Aug 9, 2023 at 8:28 AM Zhijie Hou (Fujitsu) wrote: > > On Thursday, August 3, 2023 7:30 PM Melih Mutlu > wrote: > > > Right. I attached the v26 as you asked. > > Thanks for posting the patches. > > While reviewing the patch, I noticed one rare case that it's possible that > there > are

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-08-09 Thread Michael Paquier
On Thu, Aug 10, 2023 at 10:15:40AM +0900, Kyotaro Horiguchi wrote: > At Thu, 10 Aug 2023 10:00:17 +0900 (JST), Kyotaro Horiguchi > wrote in >> At Wed, 9 Aug 2023 17:44:49 +0900, Michael Paquier >> wrote in While it's a kind of bug in total, we encountered a case where an excessively

Re: Support to define custom wait events for extensions

2023-08-09 Thread Masahiro Ikeda
Hi, Thanks for your comments about the v2 patches. I updated to v3 patches. The main changes are: * remove the AddinShmemInitLock assertion * add the new lock (WaitEventExtensionLock) to wait_event_names.txt * change "static const int wee_hash_XXX_size" to "#define XXX" * simplify worker_spi. I

Make psql's qeury canceling test simple by using signal() routine of IPC::Run

2023-08-09 Thread Yugo NAGATA
Hello, Currently, the psql's test of query cancelling (src/bin/psql/t/020_cancel.pl) gets the PPID of a running psql by using "\!" meta command, and sends SIGINT to the process by using "kill". However, IPC::Run provides signal() routine that sends a signal to a running process, so I think we can

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-08-09 Thread Amit Kapila
On Thu, Aug 10, 2023 at 6:46 AM Masahiko Sawada wrote: > > On Wed, Aug 9, 2023 at 1:15 PM Amit Kapila wrote: > > > > On Wed, Aug 9, 2023 at 8:01 AM Masahiko Sawada > > wrote: > > > > I feel it would be a good idea to provide such a tool for users to > > avoid getting errors during upgrade but I

Re: pgbnech: allow to cancel queries during benchmark

2023-08-09 Thread Yugo NAGATA
On Wed, 9 Aug 2023 11:18:43 +0200 (CEST) Fabien COELHO wrote: > > I forgot, about the test: > > I think that it should be integrated in the existing > "001_pgbench_with_server.pl" script, because a TAP script is pretty > expensive as it creates a cluster, starts it… before running the test.

Re: pgbnech: allow to cancel queries during benchmark

2023-08-09 Thread Yugo NAGATA
Hello Fabien, On Wed, 9 Aug 2023 11:06:24 +0200 (CEST) Fabien COELHO wrote: > > Hello Yugo-san, > > Some feedback about v2. > > There is some dead code (&& false) which should be removed. I forgot to remove the debug code. I'll remove it. > >>> In multi-threads, only one thread can catches

Re: Adding a LogicalRepWorker type field

2023-08-09 Thread Peter Smith
On Wed, Aug 9, 2023 at 4:18 PM Amit Kapila wrote: > > On Tue, Aug 8, 2023 at 1:39 PM Peter Smith wrote: > > > > On Fri, Aug 4, 2023 at 5:45 PM Peter Smith wrote: > > > > v4-0001 uses only 3 simple inline functions. Callers always pass > > parameters as Bharath had suggested. > > > > * > - Assert

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
FYI, the current PG 16 release notes are available at: https://momjian.us/pgsql_docs/release-16.html I plan to add markup next week. I am sorry I was away most of July so could not update this until now. -- Bruce Momjian https://momjian.us EDB

Re: 2023-08-10 release announcement draft

2023-08-09 Thread Jonathan S. Katz
On 8/8/23 11:13 PM, Robert Treat wrote: "Users who have skipped one or more update releases may need to run additional, post-update steps; " The comma should be removed. "please see the release notes for earlier versions for details." Use of 'for' twice is grammatically incorrect; I am partia

Re: 2023-08-10 release announcement draft

2023-08-09 Thread Jonathan S. Katz
On 8/9/23 1:04 AM, Noah Misch wrote: On Mon, Aug 07, 2023 at 10:03:44PM -0400, Jonathan S. Katz wrote: Fixes in PostgreSQL 16 Beta 3 - The following includes fixes included in PostgreSQL 16 Beta 3: With both "includes" and "included" there, this line reads awkwardl

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-08-09 Thread Masahiko Sawada
On Wed, Aug 9, 2023 at 1:15 PM Amit Kapila wrote: > > On Wed, Aug 9, 2023 at 8:01 AM Masahiko Sawada wrote: > > > > On Mon, Aug 7, 2023 at 6:02 PM Amit Kapila wrote: > > > > > > On Mon, Aug 7, 2023 at 2:02 PM Masahiko Sawada > > > wrote: > > > > > > > > WAL records for hint bit updates could b

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-08-09 Thread Kyotaro Horiguchi
At Thu, 10 Aug 2023 10:00:17 +0900 (JST), Kyotaro Horiguchi wrote in > At Wed, 9 Aug 2023 17:44:49 +0900, Michael Paquier > wrote in > > > While it's a kind of bug in total, we encountered a case where an > > > excessively large xl_tot_len actually came from a corrupted > > > record. [1] > >

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-08-09 Thread Kyotaro Horiguchi
At Wed, 9 Aug 2023 17:44:49 +0900, Michael Paquier wrote in > > While it's a kind of bug in total, we encountered a case where an > > excessively large xl_tot_len actually came from a corrupted > > record. [1] > > Right, I remember this one. I think that Thomas was pretty much right > that thi

Re: Fix last unitialized memory warning

2023-08-09 Thread Julien Rouhaud
On Wed, Aug 09, 2023 at 10:29:56AM -0500, Tristan Partin wrote: > On Wed Aug 9, 2023 at 10:02 AM CDT, Peter Eisentraut wrote: > > > > This patch has apparently upset one buildfarm member with a very old > > compiler: > > https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=lapwing&br=HEAD >

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > > Remove libpq support for SCM credential authentication (Michael Paquier) > > Since the point of removing it is the deep unlikelihood of anyone using it, I > wouldn't list this in "incompatibilities". I moved this to the Source Code s

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > > Author: Robert Haas > > 2022-08-25 [e3ce2de09] Allow grant-level control of role inheritance > > behavior. > > --> > > > > > > > > Allow GRANT to control role inheritance behavior (Robert Haas) > > > > > > > > By default, role

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Peter Geoghegan
On Wed, Aug 9, 2023 at 11:15 AM Tomas Vondra wrote: > Cool. I'll try to build my own set of examples that I find interesting > either because it's what the patch aims to help with, or because I > expect it to be problematic for some reason. And then we can compare. That would be great. I definite

Re: PG 16 draft release notes ready

2023-08-09 Thread Michael Paquier
On Wed, Aug 09, 2023 at 05:45:27PM -0400, Bruce Momjian wrote: > That is very helpful. I added this to the release notes Server > Configuration section: > > > > Allow huge pages to work on newer versions of Windows 10 (Thomas Munro) > > > > This adds

Should the archiver process always make sure that the timeline history files exist in the archive?

2023-08-09 Thread Jimmy Yih
Hello pgsql-hackers, While testing out some WAL archiving and PITR scenarios, it was observed that enabling WAL archiving for the first time on a primary that was on a timeline higher than 1 would not initially archive the timeline history file for the timeline it was currently on. While this migh

Re: Support to define custom wait events for extensions

2023-08-09 Thread Michael Paquier
On Wed, Aug 09, 2023 at 08:10:42PM +0900, Masahiro Ikeda wrote: > * create second hash table to find a event id from a name to > identify uniquness. It enable extensions which don't use share > memory for their use to define custom wait events because > WaitEventExtensionNew() will not alloca

Re: pg_dump needs SELECT privileges on irrelevant extension table

2023-08-09 Thread Jacob Champion
Hi all, v3 fixes a doc comment I forgot to fill in; there are no other code changes. To try to further reduce the activation energy, I've also attached an attempt at a backport to 11. The main difference is the absence of catalogIdHash, which showed up in 15, so we don't get the benefit of that de

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: > > https://momjian.us/pgsql_docs/release-16.html > > > > > > > > > > > Restrict the privileges of CREATEROLE roles (Robert Haas) > > > > > > > > Previously roles

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Sun, Jul 23, 2023 at 08:19:55PM +0900, Michael Paquier wrote: > On Tue, Jul 04, 2023 at 05:32:07PM -0400, Bruce Momjian wrote: > > On Tue, Jul 4, 2023 at 03:31:05PM +0900, Michael Paquier wrote: > >> On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: > >> Sawada-san has mentioned on

Re: Adding a pg_servername() function

2023-08-09 Thread Christoph Moench-Tegeder
## GF (phab...@gmail.com): > In the past I needed such a pg_servername() function because in a project I > was engaged on they needed to distribute "requests" directed to a in-house > management service running on network servers, and each one had a DB, so we > went with pglogical to be used as a

Re: Show WAL write and fsync stats in pg_stat_io

2023-08-09 Thread Melanie Plageman
On Thu, Aug 03, 2023 at 04:38:41PM +0300, Nazir Bilal Yavuz wrote: > Current status of the patch is: > - 'WAL read' stats in xlogrecovery.c are added to pg_stat_io. > - IOCONTEXT_INIT is added to count 'WAL init'. 'WAL init' stats are > added to pg_stat_io. > - pg_stat_io shows different op_bytes f

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Tomas Vondra
On 8/9/23 19:53, Peter Geoghegan wrote: > On Wed, Aug 9, 2023 at 10:00 AM Tomas Vondra > wrote: >> Anyway, I find this discussion rather abstract and I'll probably forget >> half the important cases by next week. Maybe it'd be good to build a set >> of examples demonstrating the interesting cases?

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Sun, Jul 23, 2023 at 02:09:17PM +0300, Pavel Luzanov wrote: > Please consider to add item to the psql section: > > Add psql \drg command to display role grants and remove the "Member of" > column from \du & \dg altogether (d65ddaca) The release notes are only current as of 2023-06-26 and I wil

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Sun, Jul 23, 2023 at 12:45:17PM +0800, jian he wrote: > > https://momjian.us/pgsql_docs/release-16.html > > > > I will adjust it to the feedback I receive; that URL will quickly show > > all updates. > > > Create a predefined role and grantable privilege with permission to > > perfor

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Fri, Jul 14, 2023 at 09:29:42PM +0200, Erik Rijkers wrote: > Op 7/4/23 om 23:32 schreef Bruce Momjian: > > > > https://momjian.us/pgsql_docs/release-16.html > > I noticed these: > > 'new new RULES' should be > 'new RULES' > > 'Perform apply of large transactions' should be > 'Perfor

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Peter Geoghegan
On Wed, Aug 9, 2023 at 10:00 AM Tomas Vondra wrote: > Anyway, I find this discussion rather abstract and I'll probably forget > half the important cases by next week. Maybe it'd be good to build a set > of examples demonstrating the interesting cases? We've already used a > couple tenk1 queries fo

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Fri, Jul 14, 2023 at 08:20:59PM +0200, Laurenz Albe wrote: > On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. > > The release notes still have: > > - Have initdb use ICU by default if ICU is enabled in the binary (Jeff Dav

Re: PG 16 draft release notes ready

2023-08-09 Thread Bruce Momjian
On Fri, Jul 14, 2023 at 08:16:38PM +0200, Laurenz Albe wrote: > On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. > > The release notes say: > > - Prevent \df+ from showing function source code (Isaac Morland) > > Function

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-09 Thread Alvaro Herrera
Hello So pginfra had a little chat about this. Firstly, there's consensus that it makes sense for pginfra to help out with some persistent workers in our existing VM system; however there are some aspects that need some further discussion, to avoid destabilizing the rest of the infrastructure. W

Re: Using defines for protocol characters

2023-08-09 Thread Alvaro Herrera
On 2023-Aug-09, Nathan Bossart wrote: > On Wed, Aug 09, 2023 at 10:44:42AM -0600, Dave Cramer wrote: > > On Wed, 9 Aug 2023 at 10:34, Tom Lane wrote: > >> I agree with Peter: let's use the names in the protocol document > >> with a single prefix. I've got mixed feelings about whether that prefix

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Tomas Vondra
On 8/8/23 22:54, Peter Geoghegan wrote: > On Tue, Aug 8, 2023 at 1:24 PM Tomas Vondra > wrote: >>> Assuming that that happens, then it immediately gives index scans a >>> huge advantage over bitmap index scans. At that point it seems >>> important to describe (in high level terms) where it is that

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Peter Geoghegan
On Wed, Aug 9, 2023 at 9:05 AM Tomas Vondra wrote: > But in the example you shared yesterday, the problem is not really about > visibility checks. In fact, the index scan costing completely ignores > the VM checks - it didn't matter before, and the patch did not change > this. It's about the numbe

Re: Using defines for protocol characters

2023-08-09 Thread Nathan Bossart
On Wed, Aug 09, 2023 at 10:44:42AM -0600, Dave Cramer wrote: > On Wed, 9 Aug 2023 at 10:34, Tom Lane wrote: >> I agree with Peter: let's use the names in the protocol document >> with a single prefix. I've got mixed feelings about whether that prefix >> should have an underscore, though. > > Wel

Re: Using defines for protocol characters

2023-08-09 Thread Dave Cramer
On Wed, 9 Aug 2023 at 10:34, Tom Lane wrote: > Dave Cramer writes: > > On Wed, 9 Aug 2023 at 09:19, Peter Eisentraut > wrote: > >> 3. IMO, the names of the protocol messages in protocol.sgml are > >> canonical. Your patch appends "Request" and "Response" in cases where > >> that is not part of

Re: [RFC] Clang plugin for catching suspicious typedef casting

2023-08-09 Thread Dmitry Dolgov
> On Wed, Aug 09, 2023 at 03:23:32PM +0200, Peter Eisentraut wrote: > On 03.08.23 18:56, Dmitry Dolgov wrote: > > I would like to get your opinion, folks. Does it sound interesting > > enough for the community, is it worth it to pursue the idea? > > I think it's interesting, and doesn't look too in

Re: Using defines for protocol characters

2023-08-09 Thread Tom Lane
Dave Cramer writes: > On Wed, 9 Aug 2023 at 09:19, Peter Eisentraut wrote: >> 3. IMO, the names of the protocol messages in protocol.sgml are >> canonical. Your patch appends "Request" and "Response" in cases where >> that is not part of the actual name. Also, some messages are documented >> to

Re: Adding a pg_servername() function

2023-08-09 Thread Tom Lane
GF writes: > On Wed, 9 Aug 2023 at 16:05, Tom Lane wrote: >> I actually do object to this, because I think the concept of "server >> name" is extremely ill-defined > But the gethostname() function is well defined, both in Linux and in > Windows. Sure, its call convention is standardized. But I

Re: Using defines for protocol characters

2023-08-09 Thread Dave Cramer
On Wed, 9 Aug 2023 at 09:19, Peter Eisentraut wrote: > 1. As was discussed, these definitions should go into > src/include/libpq/pqcomm.h, not a new file. > I'm ambivalent, this is very easy to do. > > 2. I would prefer an underscore after PgMsg, like PqMsg_DescribeRequest, > so it's easier to

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Tomas Vondra
On 8/8/23 23:03, Peter Geoghegan wrote: > On Tue, Aug 8, 2023 at 1:49 PM Tomas Vondra > wrote: >> So we expect 1250 rows. If that was accurate, the index scan would have >> to do 1250 heap fetches. It's just luck the index scan doesn't need to >> do that. I don't this there's a chance to improv

Re: pgsql: Ignore BRIN indexes when checking for HOT udpates

2023-08-09 Thread Tomas Vondra
On 8/9/23 11:11, Alvaro Herrera wrote: > On 2021-Nov-30, Tomas Vondra wrote: > >> Ignore BRIN indexes when checking for HOT udpates > > I was trying to use RelationGetIndexAttrBitmap for something and > realized that its header comment does not really explain things very > well. That was already

Re: pg15: reltuples stuck at -1 after pg_upgrade and VACUUM

2023-08-09 Thread Peter Geoghegan
On Wed, Aug 9, 2023 at 6:18 AM Tom Lane wrote: > Perhaps, though, there's a case for forcing all pages to be visited > if we start with reltuples == -1? I'm not sure it matters much > though given that you also need an ANALYZE run to be really in a > good place after pg_upgrade. The ANALYZE shou

Re: Adding a pg_servername() function

2023-08-09 Thread GF
On Wed, 9 Aug 2023 at 16:05, Tom Lane wrote: > Peter Eisentraut writes: > > On 09.08.23 08:42, Laetitia Avrot wrote: > >> My question is very simple: Do you oppose having this feature in > Postgres? > > > I think this is pretty harmless(*) and can be useful, so it seems > > reasonable to pursue.

Re: Fix last unitialized memory warning

2023-08-09 Thread Tristan Partin
On Wed Aug 9, 2023 at 10:02 AM CDT, Peter Eisentraut wrote: On 09.08.23 10:07, Peter Eisentraut wrote: > On 08.08.23 17:14, Tristan Partin wrote: >>> I was able to reproduce the warning now on Fedora.  I agree with the >>> patch >>> >>> -   PgBenchValue vargs[MAX_FARGS]; >>> +   PgBench

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-09 Thread David Rowley
On Thu, 10 Aug 2023 at 03:12, Ashutosh Bapat wrote: > Thinking more about it, I think memory used is the only right metrics. > It's an optimization in MemoryContext implementation that malloc'ed > memory is not freed when it is returned using free(). I guess it depends on the problem you're tryin

Re: Using defines for protocol characters

2023-08-09 Thread Peter Eisentraut
1. As was discussed, these definitions should go into src/include/libpq/pqcomm.h, not a new file. 2. I would prefer an underscore after PgMsg, like PqMsg_DescribeRequest, so it's easier to visually locate the start of the actual message name. 3. IMO, the names of the protocol messages in prot

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-09 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 8:09 PM Ashutosh Bapat wrote: > > Hi David, > Replying to your comments on this thread. > > > On Tue, Aug 8, 2023 at 11:40 AM Ashutosh Bapat > > wrote: > >> > >> Hi All, > >> I have been looking at memory consumed when planning a partitionwise join > >> [1], [2] and [3].

Re: [BackendXidGetPid] only access allProcs when xid matches

2023-08-09 Thread Junwang Zhao
On Wed, Aug 9, 2023 at 10:46 PM Ashutosh Bapat wrote: > > On Wed, Aug 9, 2023 at 9:30 AM Junwang Zhao wrote: > > > > In function `BackendXidGetPid`, when looping every proc's > > TransactionId, there is no need to access its PGPROC since there > > is shared memory access: `arrayP->pgprocn

Re: Fix last unitialized memory warning

2023-08-09 Thread Peter Eisentraut
On 09.08.23 10:07, Peter Eisentraut wrote: On 08.08.23 17:14, Tristan Partin wrote: I was able to reproduce the warning now on Fedora.  I agree with the patch -   PgBenchValue vargs[MAX_FARGS]; +   PgBenchValue vargs[MAX_FARGS] = { 0 }; I suggest to also do   typedef enum   { -  

Re: [BackendXidGetPid] only access allProcs when xid matches

2023-08-09 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 9:30 AM Junwang Zhao wrote: > > In function `BackendXidGetPid`, when looping every proc's > TransactionId, there is no need to access its PGPROC since there > is shared memory access: `arrayP->pgprocnos[index]`. > > Though the compiler can optimize this kind of i

Re: Support to define custom wait events for extensions

2023-08-09 Thread Andres Freund
Hi, On 2023-08-09 20:10:42 +0900, Masahiro Ikeda wrote: > * Is there any way to not force extensions that don't use shared > memory for their use like dblink to acquire AddinShmemInitLock?; I think the caller shouldn't need to do deal with AddinShmemInitLock at all. Greetings, Andres Freund

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-09 Thread Ashutosh Bapat
Hi David, Replying to your comments on this thread. > On Tue, Aug 8, 2023 at 11:40 AM Ashutosh Bapat > wrote: >> >> Hi All, >> I have been looking at memory consumed when planning a partitionwise join >> [1], [2] and [3]. I am using the attached patch to measure the memory >> consumption. The

Re: Remove distprep

2023-08-09 Thread Andres Freund
Hi, Thanks for sending the -packagers email Peter! On 2023-08-09 16:25:28 +0200, Christoph Berg wrote: > Re: Tom Lane > > Meh ... the fact that it works fine for you doesn't mean it will work > > fine elsewhere. Since we're trying to get out from under maintaining > > the autoconf build system,

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-09 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 10:12 AM David Rowley wrote: > > On Fri, 28 Jul 2023 at 02:06, Ashutosh Bapat > wrote: > > 0001 - to measure memory consumption during planning. This is the same > > one as attached to [1]. > I have started a separate thread to discuss this patch. I am taking this discussi

Re: Remove distprep

2023-08-09 Thread Christoph Berg
Re: Tom Lane > Meh ... the fact that it works fine for you doesn't mean it will work > fine elsewhere. Since we're trying to get out from under maintaining > the autoconf build system, I don't think it makes sense to open > ourselves up to having to do more work on it. A policy of benign > neglec

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-09 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 8:14 AM Richard Guo wrote: > > With this patch, the reparameterize_path_by_child work is postponed > until createplan time, so in create_nestloop_path() the inner path is > still parameterized by top parent. So we have to check against the top > parent of outer rel. > Your

Re: Separate memory contexts for relcache and catcache

2023-08-09 Thread David Rowley
On Thu, 10 Aug 2023 at 01:23, Alvaro Herrera wrote: > > On 2023-Aug-09, Melih Mutlu wrote: > > > --Patch > > name | used_bytes | free_bytes | total_bytes > > ---+++- > > RelCacheMemoryContext |4706464 |3682144 |

Re: Remove distprep

2023-08-09 Thread Tom Lane
Christoph Berg writes: > Most notably, we are also rebuilding "configure" using autoconf 2.71 > without issues. Perhaps we can get rid of the 2.69 hardcoding there? Meh ... the fact that it works fine for you doesn't mean it will work fine elsewhere. Since we're trying to get out from under main

Re: Adding a pg_servername() function

2023-08-09 Thread Tom Lane
Peter Eisentraut writes: > On 09.08.23 08:42, Laetitia Avrot wrote: >> My question is very simple: Do you oppose having this feature in Postgres? > I think this is pretty harmless(*) and can be useful, so it seems > reasonable to pursue. I actually do object to this, because I think the concept

Re: [RFC] Clang plugin for catching suspicious typedef casting

2023-08-09 Thread Peter Eisentraut
On 03.08.23 18:56, Dmitry Dolgov wrote: I would like to get your opinion, folks. Does it sound interesting enough for the community, is it worth it to pursue the idea? I think it's interesting, and doesn't look too invasive. Maybe we can come up with three or four interesting use cases and try

Re: Separate memory contexts for relcache and catcache

2023-08-09 Thread Alvaro Herrera
On 2023-Aug-09, Melih Mutlu wrote: > --Patch > name | used_bytes | free_bytes | total_bytes > ---+++- > RelCacheMemoryContext |4706464 |3682144 | 8388608 > CatCacheMemoryContext |3489384 | 770712 |

Re: pg15: reltuples stuck at -1 after pg_upgrade and VACUUM

2023-08-09 Thread Tom Lane
Peter Geoghegan writes: > On Tue, Aug 8, 2023 at 8:43 PM Justin Pryzby wrote: >> The problem isn't that reltuples == -1 after the upgrade (which is >> normal). The issue is that if VACUUM skips all the pages, it can leave >> reltuples -1. My expectation is that after running "vacuum", no tables

Re: initial pruning in parallel append

2023-08-09 Thread Amit Langote
On Wed, Aug 9, 2023 at 9:48 PM Robert Haas wrote: > On Wed, Aug 9, 2023 at 6:22 AM Amit Langote wrote: > > > > I'm assuming it's not > > > > too ugly if ExecInitAppend() uses IsParallelWorker() to decide whether > > > > it should be writing to EState.es_part_prune_results or reading from > > > >

Re: initial pruning in parallel append

2023-08-09 Thread Robert Haas
On Wed, Aug 9, 2023 at 6:22 AM Amit Langote wrote: > > > I'm assuming it's not > > > too ugly if ExecInitAppend() uses IsParallelWorker() to decide whether > > > it should be writing to EState.es_part_prune_results or reading from > > > it -- the former if in the leader and the latter in a worker.

Re: Separate memory contexts for relcache and catcache

2023-08-09 Thread Andy Fan
> > Most catcache and relcache entries (other than index info etc.) currently > go straight into CacheMemoryContext. And I believe these two caches can be > the ones with the largest contribution to the memory usage of > CacheMemoryContext most of the time. For example, in cases where we have > lot

Separate memory contexts for relcache and catcache

2023-08-09 Thread Melih Mutlu
Hi hackers, Most catcache and relcache entries (other than index info etc.) currently go straight into CacheMemoryContext. And I believe these two caches can be the ones with the largest contribution to the memory usage of CacheMemoryContext most of the time. For example, in cases where we have lo

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 9 Aug 2023 at 20:15, Yuya Watari wrote: > I agree with your opinion that my patch lacks some explanations, so I > will consider adding more comments. However, I received the following > message from David in March. > > On Thu, Mar 9, 2023 at 6:23 AM David Rowley wrote: > > For the main pa

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-08-09 Thread Christoph Berg
Re: To Thomas Munro > 603 iterations later it hit again, but didn't log anything. (I believe > I did run "make" in the right directory.) Since that didn't seem right I'm running the tests again. There are XXX lines in the output, but it hasn't hit yet. Christoph

Re: POC, WIP: OR-clause support for indexes

2023-08-09 Thread Alena Rybakina
Hi! Thank you for your research, I'm sure it will help me to fix the problem of calculating selectivity faster) I'm sorry I didn't answer right away, to be honest, I had a full diary of urgent matters at work. For this reason, I didn't have enough time to work on this patch properly. The opti

Re: Support to define custom wait events for extensions

2023-08-09 Thread Masahiro Ikeda
Hi, Thanks for your comments to v1 patch. I made v2 patch. Main changes are * change to NAMEDATALEN * change to use dynahash from dshash * remove worker_spi_init() * create second hash table to find a event id from a name to identify uniquness. It enable extensions which don't use share memo

Re: cataloguing NOT NULL constraints

2023-08-09 Thread Alvaro Herrera
On 2023-Aug-09, Peter Eisentraut wrote: > I wonder whether the root of these problems is that we mix together primary > key constraints and not-null constraints. I understand that right now, with > the proposed patch, when a table inherits from a parent table with a primary > key constraint, we g

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-08-09 Thread Christoph Berg
Re: Thomas Munro > On Wed, Aug 9, 2023 at 2:01 AM Christoph Berg wrote: > > Putting that patch on top of v8 made it pass 294 times before exiting > > like this: > > > > [08:52:34.134](0.032s) ok 1 - buffer pin conflict: cursor with conflicting > > pin established > > Waiting for replication conn

Re: Adding a pg_servername() function

2023-08-09 Thread GF
Hi everybody, On 09.08.23 08:42, Laetitia Avrot wrote: > I agree that the feature I'm suggesting could be done with a few tricks. > I meant to simplify the life of the user by providing a simple new > feature. (Also, I might have trust issues with DNS due to several past > production disasters.)

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 9 Aug 2023 at 22:28, David Rowley wrote: > i.e: > > + Bitmapset *matching_ems = NULL; > + memcpy(&idx_iter, iter, sizeof(EquivalenceMemberIterator)); > + memcpy(&noidx_iter, iter, sizeof(EquivalenceMemberIterator)); > + > + idx_iter.use_index = true; > + noidx_iter.use_index = false; > + >

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 5 Jul 2023 at 21:58, Yuya Watari wrote: > > Hello, > > On Fri, Mar 10, 2023 at 5:38 PM Yuya Watari wrote: > > Thank you for pointing it out. I have attached the rebased version to > > this email. > > Recent commits, such as a8c09daa8b [1], have caused conflicts and > compilation errors in

Re: initial pruning in parallel append

2023-08-09 Thread Amit Langote
On Tue, Aug 8, 2023 at 11:16 PM Robert Haas wrote: > On Tue, Aug 8, 2023 at 2:58 AM Amit Langote wrote: > > Or we could consider something like the patch I mentioned in my 1st > > email. The idea there was to pass the pruning result via a separate > > channel, not the DSM chunk linked into the P

Re: logical decoding issue with concurrent ALTER TYPE

2023-08-09 Thread Amit Kapila
On Sun, Aug 6, 2023 at 11:06 AM Masahiko Sawada wrote: > > A colleague Drew Callahan (in CC) has discovered that the logical > decoding doesn't handle syscache invalidation messages properly that > are generated by other transactions. Here is example (I've attached a > patch for isolation test), >

Re: Remove distprep

2023-08-09 Thread Christoph Berg
Re: Tom Lane > Have we yet run this concept past the packagers list? I'm still > wondering if they will raise any objection to getting rid of all > the prebuilt files. No problem for Debian, we are building snapshot releases from plain git already without issues. In fact, there are already some e

Re: pgsql: Ignore BRIN indexes when checking for HOT udpates

2023-08-09 Thread Alvaro Herrera
On 2021-Nov-30, Tomas Vondra wrote: > Ignore BRIN indexes when checking for HOT udpates I was trying to use RelationGetIndexAttrBitmap for something and realized that its header comment does not really explain things very well. That was already the case before this commit, but it (this commit) d

Re: pgbnech: allow to cancel queries during benchmark

2023-08-09 Thread Fabien COELHO
Hello Yugo-san, Some feedback about v2. There is some dead code (&& false) which should be removed. Maybe it should check that cancel is not NULL before calling PQcancel? I think this is already checked as below, but am I missing something? + if (all_state[i].cancel != NULL) +

Re: Avoid stack frame setup in performance critical routines using tail calls

2023-08-09 Thread David Rowley
On Fri, 21 Jul 2023 at 14:03, David Rowley wrote: > I'll reply back with a more detailed review next week. Here's a review of v2-0001: 1. /* * XXX: Should this also be moved into alloc()? We could possibly avoid * zeroing in some cases (e.g. if we used mmap() ourselves. */ MemSetAligned(ret, 0,

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-08-09 Thread Michael Paquier
On Wed, Aug 09, 2023 at 05:00:49PM +0900, Kyotaro Horiguchi wrote: > Looks fine. Okay, I've updated the patch in consequence. I'll look at 0001 again at the beginning of next week. > While it's a kind of bug in total, we encountered a case where an > excessively large xl_tot_len actually came fr

Re: Row pattern recognition

2023-08-09 Thread Tatsuo Ishii
Attached is the v4 patch. Differences from previous patch include: > - PERMUTE is still misspelled as PREMUTE Fixed. > - PATTERN variables do not have to exist in the DEFINE clause. They are > - considered TRUE if not present. Fixed. Moreover new regression test case is added. - It was possib

Re: Adding a pg_servername() function

2023-08-09 Thread Peter Eisentraut
On 09.08.23 08:42, Laetitia Avrot wrote: I agree that the feature I'm suggesting could be done with a few tricks. I meant to simplify the life of the user by providing a simple new feature. (Also, I might have trust issues with DNS due to several past production disasters.) My question is ver

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread Yuya Watari
Hello Andrey, Ashutosh, and David, Thank you for your reply and for reviewing the patch. On Mon, Aug 7, 2023 at 5:51 PM Andrey Lepikhov wrote: > One more thing: I think, you should add comments to > add_child_rel_equivalences() and add_child_join_rel_equivalences() > on replacing of: > > if (bms

Re: Fix last unitialized memory warning

2023-08-09 Thread Peter Eisentraut
On 08.08.23 17:14, Tristan Partin wrote: I was able to reproduce the warning now on Fedora.  I agree with the patch -   PgBenchValue vargs[MAX_FARGS]; +   PgBenchValue vargs[MAX_FARGS] = { 0 }; I suggest to also do   typedef enum   { -   PGBT_NO_VALUE, +   PGBT_NO_VALUE = 0,

Re: WIP: new system catalog pg_wait_event

2023-08-09 Thread Michael Paquier
On Tue, Aug 08, 2023 at 10:16:37AM +0200, Drouvot, Bertrand wrote: > Please find attached v3 adding the wait event types. +-- There will surely be at least 9 wait event types, 240 wait events and at +-- least 27 related to WAL +select count(distinct(wait_event_type)) > 8 as ok_type, + count(

Re: cataloguing NOT NULL constraints

2023-08-09 Thread Peter Eisentraut
On 05.08.23 21:50, Dean Rasheed wrote: Anyway, I was at the same time fixing the other problem you reported with inheritance (namely, adding a PK ends up with the child column being marked NOT NULL but no corresponding constraint). At some point I wondered if the easy way out wouldn't be to give

Re: Extract numeric [field] in JSONB more effectively

2023-08-09 Thread jian he
On Wed, Aug 9, 2023 at 4:30 AM Chapman Flack wrote: > > Hi, > > Looking at the most recent patch, so far I have a minor > spelling point, and a question (which I have not personally > explored). > > The minor spelling point, the word 'field' has been spelled > 'filed' throughout this comment (just

  1   2   >