Re: Add assertion on held AddinShmemInitLock in GetNamedLWLockTranche()

2023-07-27 Thread Bharath Rupireddy
On Fri, Jul 28, 2023 at 8:54 AM Michael Paquier wrote: > > Hi all, > > While digging into the LWLock code, I have noticed that > GetNamedLWLockTranche() assumes that its caller should hold the LWLock > AddinShmemInitLock to prevent any kind of race conditions when > initializing shmem areas, but

Re: Support worker_spi to execute the function dynamically.

2023-07-27 Thread Bharath Rupireddy
On Fri, Jul 28, 2023 at 10:15 AM Michael Paquier wrote: > > Having each bgworker on its own schema would be enough to prevent > conflicts, but I'd like to add a second thing: a check on > pg_stat_activity.wait_event after starting the workers. I have added > something like that in the patch I

Re: Support worker_spi to execute the function dynamically.

2023-07-27 Thread Michael Paquier
On Thu, Jul 27, 2023 at 07:23:32PM -0700, Andres Freund wrote: > As written, dynamic and static workers race each other. It doesn't make a lot > of sense to me to use the same ids for either? > > The attached patch reproduces the problem on master. > > Note that without the sleep(3) in the test

Re: Postgres v15 windows bincheck regression test failures

2023-07-27 Thread Alexander Lakhin
Hello, 28.07.2023 05:17, Noah Misch wrote: On Tue, Jun 20, 2023 at 07:49:52AM -0400, Russell Foster wrote: /* * We don't use Unix-domain sockets on Windows by default, even if the * build supports them. (See comment at remove_temp() for a reason.) * Override at your own risk. */ Is there

Add assertion on held AddinShmemInitLock in GetNamedLWLockTranche()

2023-07-27 Thread Michael Paquier
Hi all, While digging into the LWLock code, I have noticed that GetNamedLWLockTranche() assumes that its caller should hold the LWLock AddinShmemInitLock to prevent any kind of race conditions when initializing shmem areas, but we don't make sure that's the case. The sole caller of

Re: Support worker_spi to execute the function dynamically.

2023-07-27 Thread Andres Freund
Hi, The new test fails with my AIO branch occasionally. But I'm fairly certain that's just due to timing differences. Excerpt from the log: 2023-07-27 21:43:00.385 UTC [42339] LOG: worker_spi worker 3 initialized with schema3.counted 2023-07-27 21:43:00.399 UTC [42344] 001_worker_spi.pl LOG:

Re: Postgres v15 windows bincheck regression test failures

2023-07-27 Thread Noah Misch
On Tue, Jun 20, 2023 at 07:49:52AM -0400, Russell Foster wrote: > /* > * We don't use Unix-domain sockets on Windows by default, even if the > * build supports them. (See comment at remove_temp() for a reason.) > * Override at your own risk. > */ > > Is there some sort of race condition in the

Re: Support to define custom wait events for extensions

2023-07-27 Thread Michael Paquier
On Thu, Jul 27, 2023 at 06:29:22PM +0900, Masahiro Ikeda wrote: > I suspect that I forgot to specify "volatile" to the variable > for the spinlock. + if (!IsUnderPostmaster) + { + /* Allocate space in shared memory. */ + waitEventExtensionCounter = (WaitEventExtensionCounter *) +

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-27 Thread Peter Smith
On Thu, Jul 27, 2023 at 11:30 PM Melih Mutlu wrote: > > Hi Peter, > > Peter Smith , 26 Tem 2023 Çar, 07:40 tarihinde şunu > yazdı: >> >> Here are some comments for patch v22-0001. >> >> == >> 1. General -- naming conventions >> >> There is quite a lot of inconsistency with variable/parameter

add timing information to pg_upgrade

2023-07-27 Thread Nathan Bossart
I've been looking into some options for reducing the amount of downtime required for pg_upgrade, and $SUBJECT seemed like something that would be worthwhile independent of that effort. The attached work-in-progress patch adds the elapsed time spent in each step, which looks like this:

Re: Removing the fixed-size buffer restriction in hba.c

2023-07-27 Thread Michael Paquier
On Thu, Jul 27, 2023 at 12:11:38PM -0400, Tom Lane wrote: > Done that way. Thanks for looking at it! Thanks for applying! -- Michael signature.asc Description: PGP signature

Re: Removing the fixed-size buffer restriction in hba.c

2023-07-27 Thread Tom Lane
Michael Paquier writes: > Seeing what this code does, the odds of needing an err_msg seem rather > low to me, so I would just remove it for now for the sake of clarity. > It can always be added later if need be. Done that way. Thanks for looking at it! regards, tom lane

Re: POC: Extension for adding distributed tracing - pg_tracing

2023-07-27 Thread Aleksander Alekseev
Hi, > Also FYI, there are build warnings because functions > const char * get_span_name(const Span * span, const char *qbuffer) > and > const char * get_operation_name(const Span * span, const char *qbuffer) > do not have default inside switch and no return outside of switch. You are right,

Re: POC: Extension for adding distributed tracing - pg_tracing

2023-07-27 Thread Nikita Malakhov
Hi, Also FYI, there are build warnings because functions const char * get_span_name(const Span * span, const char *qbuffer) and const char * get_operation_name(const Span * span, const char *qbuffer) do not have default inside switch and no return outside of switch. -- Regards, Nikita Malakhov

Re: [May be a bug] double free or corruption

2023-07-27 Thread Ashutosh Bapat
Hi Wen, On Wed, Jul 26, 2023 at 7:55 PM Wen Yi wrote: ... snip ... > - Backtrace - > corrupted double-linked list > > > Fatal signal: Aborted > - Backtrace - > done > server started > 0x5557bf5908b0 ??? ... snip ... > - > A fatal error internal to GDB has

Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-07-27 Thread Ashutosh Bapat
Hi All, In try_partitionwise_join() we create SpecialJoinInfo structures for child joins by translating SpecialJoinInfo structures applicable to the parent join. These SpecialJoinInfos are not used outside try_partitionwise_join() but we do not free memory allocated to those. In

Memory consumed by paths during partitionwise join planning

2023-07-27 Thread Ashutosh Bapat
Hi All, If add_path() and add_partial_path() do not find a new path to be superior to any existing paths, they free the new path. They free an existing path if it is found to be inferior to the new path. But not all paths surviving in a RelOptInfo are used to create paths for relations which use

Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-07-27 Thread Ashutosh Bapat
Hi All, Following up on [1] ... A restrictlist is a list of RestrictInfo nodes, each representing one clause, applicable to a set of joins. When partitionwise join is used as a strategy, the restrictlists for child join are obtained by translating the restrictlists for the parent in

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-27 Thread Peter Geoghegan
On Thu, Jul 27, 2023 at 7:59 AM Matthias van de Meent wrote: > > Basically, the patch that added that feature had to revise the index > > AM API, in order to support a mode of operation where scans return > > groupings rather than tuples. Whereas this patch requires none of > > that. It makes

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-27 Thread Matthias van de Meent
On Thu, 27 Jul 2023 at 06:14, Peter Geoghegan wrote: > > On Wed, Jul 26, 2023 at 12:07 PM Matthias van de Meent > wrote: > > We could cache the last accessed leaf page across amrescan operations > > to reduce the number of index traversals needed when the join key of > > the left side is highly

Memory consumption during partitionwise join planning

2023-07-27 Thread Ashutosh Bapat
Hi All, When we implemented partitionwise join we disabled it by default (by setting enable_partitionwise_join to false by default) since it consumed a lot of memory and took a lot of time [1]. We also set enable_partitionwise_aggregate to false by default since partitionwise aggregates require

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-27 Thread Melih Mutlu
Hi Peter, Peter Smith , 26 Tem 2023 Çar, 07:40 tarihinde şunu yazdı: > Here are some comments for patch v22-0001. > > == > 1. General -- naming conventions > > There is quite a lot of inconsistency with variable/parameter naming > styles in this patch. I understand in most cases the names

Re: Let's make PostgreSQL multi-threaded

2023-07-27 Thread David Geier
Hi, On 6/7/23 23:37, Andres Freund wrote: I think we're starting to hit quite a few limits related to the process model, particularly on bigger machines. The overhead of cross-process context switches is inherently higher than switching between threads in the same process - and my suspicion is

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-27 Thread Matthias van de Meent
On Wed, 26 Jul 2023 at 15:42, Peter Geoghegan wrote: > > On Wed, Jul 26, 2023 at 5:29 AM Matthias van de Meent > > I'm not sure I understand. MDAM seems to work on an index level to > > return full ranges of values, while "skip scan" seems to try to allow > > systems to signal to the index to

Re: Issue in postgres_fdw causing unnecessary wait for cancel request reply

2023-07-27 Thread Kuwamura Masaki
Hi, Fujii-san > Regarding the WARNING message, another idea is to pass the return value > of PQgetCancel() directly to PQcancel() as follows. If NULL is passed, > PQcancel() will detect it and set the proper error message to errbuf. > Then the warning message "WARNING: could not send cancel

Re: POC: Extension for adding distributed tracing - pg_tracing

2023-07-27 Thread Nikita Malakhov
Hi! I've tried to test the extension, but got errors calling make check and cannot install the extension. I've applied the patch onto current master and configured it as: ./configure --enable-debug --enable-cassert --enable-depend --enable-tap-tests Could you please advise if I'm doing something

Re: incremental-checkopints

2023-07-27 Thread Tomas Vondra
On 7/26/23 21:53, Matthias van de Meent wrote: > On Wed, 26 Jul 2023 at 20:58, Tomas Vondra > wrote: >> >> >> >> On 7/26/23 15:16, Matthias van de Meent wrote: >>> On Wed, 26 Jul 2023 at 14:41, Alvaro Herrera >>> wrote: Hello On 2023-Jul-26, Thomas wen wrote: > Hi

RE: remaining sql/json patches

2023-07-27 Thread Shinoda, Noriyoshi (HPE Services Japan - FSIP)
Hi, Thank you for developing such a great feature. The attached patch formats the documentation like any other function definition: - Added right parenthesis to json function calls. - Added to json functions. - Added a space to the 'expression' part of the json_scalar function. - Added a space

Re: Support to define custom wait events for extensions

2023-07-27 Thread Masahiro Ikeda
Hi, all. Sorry for late reply. I am still mid-way through the review of the core APIs, but attached is my current version in progress, labelled v8. I'll continue tomorrow. I'm aware of some typos in the commit message of this patch, and the dynamic bgworker launch is failing in the CI for

Re: Support to define custom wait events for extensions

2023-07-27 Thread Michael Paquier
On Wed, Jul 19, 2023 at 10:57:39AM -0500, Tristan Partin wrote: > > + > > +Custom Wait Events for Add-ins > > This would be the second use of "Add-ins" ever, according to my search. > Should this be "Extensions" instead? Yes, I would think that just "Custom Wait Events" is enough here.

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-27 Thread David Rowley
On Thu, 27 Jul 2023 at 14:51, David Rowley wrote: > Just to keep this moving and to make it easier for people to test the > pg_strtoint patches, I've pushed the fix_COPY_DEFAULT.patch patch. > The only thing I changed was to move the line that was allocating the > array to a location more aligned

Improvements in pg_dump/pg_restore toc format and performances

2023-07-27 Thread Pierre Ducroquet
Hi Following the thread "Inefficiency in parallel pg_restore with many tables", I started digging into why the toc.dat files are that big and where time is spent when parsing them. I ended up writing several patches that shaved some time for pg_restore -l, and reduced the toc.dat size. First

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-07-27 Thread David Steele
Hi Thomas, On 7/26/23 06:06, Thomas Munro wrote: While chatting to Robert and Andres about all this, a new idea came up. Or, rather, one of the first ideas that was initially rejected, now resurrected to try out a suggestion of Andres’s on how to de-pessimise it. Unfortunately, it also

Re: [PoC] Reducing planning time when tables have many partitions

2023-07-27 Thread Andrey Lepikhov
On 5/7/2023 16:57, Yuya Watari wrote: Hello, On Fri, Mar 10, 2023 at 5:38 PM Yuya Watari wrote: Thank you for pointing it out. I have attached the rebased version to this email. Recent commits, such as a8c09daa8b [1], have caused conflicts and compilation errors in these patches. I have

Re: Synchronizing slots from primary to standby

2023-07-27 Thread Amit Kapila
On Wed, Jul 26, 2023 at 5:55 PM shveta malik wrote: > > On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy > wrote: > > > > On Fri, Jul 21, 2023 at 5:16 PM shveta malik wrote: > > > > > > Thanks Bharat for letting us know. It is okay to split the patch, it > > > may definitely help to understand

Re: Synchronizing slots from primary to standby

2023-07-27 Thread shveta malik
On Thu, Jul 27, 2023 at 10:55 AM Amit Kapila wrote: > > On Wed, Jul 26, 2023 at 10:31 AM shveta malik wrote: > > > > On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila wrote: > > > > > > On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy > > > wrote: > > > > > > > > Is having one (or a few more - not