Re: Support logical replication of DDLs

2022-10-31 Thread Peter Smith
Here are some more comments for the patch v32-0001, file: src/backend/commands/ddl_deparse.c This is a WIP, it being such a large file... == 1. General - comments For better consistency, I suggest using uppercase for all the single-line comments in the function bodies. There are multiple

Check return value of pclose() correctly

2022-10-31 Thread Peter Eisentraut
I noticed that some (not all) callers didn't check the return value of pclose() or ClosePipeStream() correctly. Either they didn't check it at all or they treated it like the return of fclose(). Here is a patch with fixes. (A failure to run the command issued by popen() is usually reported

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Dilip Kumar
On Mon, Oct 31, 2022 at 11:03 AM Kyotaro Horiguchi wrote: > > At Thu, 27 Oct 2022 09:59:14 -0700, Andres Freund wrote > in > > But I think we can solve that fairly reasonably nonetheless. We can change > > PGPROC->lwWaiting to not just be a boolean, but have three states: > > 0: not waiting > >

Re: Expand palloc/pg_malloc API

2022-10-31 Thread Peter Eisentraut
On 11.10.22 18:04, Tom Lane wrote: Peter Eisentraut writes: On 14.09.22 06:53, Tom Lane wrote: Actually ... an even-more-terrifyingly-plausible misuse is that the supplied oldsize is different from the actual previous allocation. We should try to check that. In MEMORY_CONTEXT_CHECKING builds

Re: Segfault on logical replication to partitioned table with foreign children

2022-10-31 Thread Dilip Kumar
On Sun, Oct 30, 2022 at 7:09 PM Tom Lane wrote: > > Dilip Kumar writes: > > Yes, this looks like a bug and your fix seems correct to me. It would > > be nice to add a test case for this scenario. > > A test case doesn't seem that exciting to me. If we were trying to > make it actually work,

Re: Fast COPY FROM based on batch insert

2022-10-31 Thread Etsuro Fujita
On Fri, Oct 28, 2022 at 7:53 PM Andrey Lepikhov wrote: > On 28/10/2022 16:12, Etsuro Fujita wrote: > > I think there is another patch that improves performance of COPY FROM > > for foreign tables using COPY FROM STDIN, but if Andrey (or anyone > > else) want to work on it again, I think it would

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2022-10-31 Thread Bharath Rupireddy
On Mon, Oct 31, 2022 at 5:01 AM Michael Paquier wrote: > > On Sun, Oct 30, 2022 at 03:44:32PM +0100, Alvaro Herrera wrote: > > So I'm kinda proposing that we only do the forward struct initialization > > dance when it really saves on things -- in particular, when it helps > > avoid or reduce

Re: Adding doubly linked list type which stores the number of items in the list

2022-10-31 Thread Bharath Rupireddy
On Mon, Oct 31, 2022 at 8:26 AM David Rowley wrote: > > I looked at dlist_check() and I didn't quite manage to figure out why > the cast is needed. As far as I can see, there are no calls where we > only pass dlist_head solely for the dlist_check(). For > dlist_member_check(), dlist_delete_from()

Re: Lock on ShmemVariableCache fields?

2022-10-31 Thread Zhang Mingli
HI, On Oct 31, 2022, 10:48 +0800, Japin Li , wrote: > > Hi, hackers > > The VariableCacheData says nextOid and oidCount are protected by > OidGenLock. However, we update them without holding the lock on > OidGenLock in BootStrapXLOG(). Same as nextXid, for other fields > that are protected by

Re: GUC values - recommended way to declare the C variables?

2022-10-31 Thread Peter Smith
On Mon, Oct 31, 2022 at 4:02 PM Michael Paquier wrote: > > On Mon, Oct 31, 2022 at 12:01:33PM +1100, Peter Smith wrote: > > SUGGESTION > > /* Only applicable when prefetching is available */ > > Thanks for the suggestion. Done this way, then. > > > +/* Disabled on Windows as the performance

Re: Adding doubly linked list type which stores the number of items in the list

2022-10-31 Thread David Rowley
On Mon, 31 Oct 2022 at 19:05, Bharath Rupireddy wrote: > So, when an overflow occurs, the head->count wraps around after > PG_UINT32_MAX, meaning, becomes 0 and we will catch it in an assert > build. This looks reasonable to me. However, the responsibility lies > with the developers to deal with

Re: Lock on ShmemVariableCache fields?

2022-10-31 Thread Japin Li
On Mon, 31 Oct 2022 at 14:15, Zhang Mingli wrote: > HI, > > On Oct 31, 2022, 10:48 +0800, Japin Li , wrote: >> >> I also find a similar code in StartupXLOG(). Why we don't hold the lock >> on OidGenLock when updating ShmemVariableCache->nextOid and >> ShmemVariableCache->oidCount? >> >> If the

Re: Proposal to use JSON for Postgres Parser format

2022-10-31 Thread Alexander Korotkov
On Fri, Oct 28, 2022 at 4:27 PM Andrew Dunstan wrote: > On 2022-10-27 Th 19:38, Andres Freund wrote: > > Hi, > > > > On 2022-09-19 22:29:15 -0400, Tom Lane wrote: > >> Maybe a compromise could be found whereby we provide a conversion function > >> that converts whatever the catalog storage format

Re: shared memory stats ideas

2022-10-31 Thread Drouvot, Bertrand
Hi, On 10/20/22 9:17 AM, Drouvot, Bertrand wrote: On 10/19/22 8:19 PM, Andres Freund wrote: 2) Split index and table statistics into different types of stats We track both types of statistics in the same format and rename column in views etc to make them somewhat sensible. A number of the

Re: doc: Fix description of how the default user name is chosen

2022-10-31 Thread Peter Eisentraut
On 09.07.22 17:52, David G. Johnston wrote: No.  It is always the user parameter.  It just so happens that parameter also has a default.  And so while there is a transitive aspect the resolution of the user parameter happens first, using the OS user if needed, then the dbname parameter is

Re: Prefetch the next tuple's memory during seqscans

2022-10-31 Thread Aleksander Alekseev
Hi David, > I'll add this to the November CF. Thanks for the patch. I wonder if we can be sure and/or check that there is no performance degradation under different loads and different platforms... Also I see 0001 and 0003 but no 0002. Just wanted to double check that there is no patch

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Bharath Rupireddy
On Thu, Oct 27, 2022 at 10:29 PM Andres Freund wrote: > > But I think we can solve that fairly reasonably nonetheless. We can change > PGPROC->lwWaiting to not just be a boolean, but have three states: > 0: not waiting > 1: waiting in waitlist > 2: waiting to be woken up > > which we then can use

Re: Adding doubly linked list type which stores the number of items in the list

2022-10-31 Thread Bharath Rupireddy
On Mon, Oct 31, 2022 at 12:44 PM David Rowley wrote: > > On Mon, 31 Oct 2022 at 19:05, Bharath Rupireddy > wrote: > > So, when an overflow occurs, the head->count wraps around after > > PG_UINT32_MAX, meaning, becomes 0 and we will catch it in an assert > > build. This looks reasonable to me.

Re: warn if GUC set to an invalid shared library

2022-10-31 Thread Justin Pryzby
On Sun, Oct 30, 2022 at 04:12:33PM -0700, Maciek Sakrejda wrote: > On Sat, Oct 29, 2022 at 10:40 AM Justin Pryzby wrote: > > > > On Fri, Sep 02, 2022 at 05:24:58PM -0500, Justin Pryzby wrote: > > > It caused no issue when I changed: > > > > > > /* Check that it's

Commitfest documentation

2022-10-31 Thread Jehan-Guillaume de Rorthais
Hi, In the commitfest application, I was wondering today what was the exact meaning and difference between open/closed status (is it only for the current commitfest?) and between «waiting for author» and «Returned with feedback». I couldn't find a clear definition searching the wiki, the mailing

Re: Tracking last scan time

2022-10-31 Thread Dave Page
FYI, this is not intentional, and I do plan to look into it, however I've been somewhat busy with pgconfeu, and am travelling for the rest of this week as well. On Sun, 23 Oct 2022 at 21:09, Robert Treat wrote: > On Fri, Oct 14, 2022 at 2:55 PM Dave Page wrote: > > On Fri, 14 Oct 2022 at

Re: [patch] Have psql's \d+ indicate foreign partitions

2022-10-31 Thread Ian Lawrence Barwick
2022年10月27日(木) 16:12 Alvaro Herrera : > > On 2022-Oct-24, Justin Pryzby wrote: > > > On Mon, Oct 24, 2022 at 09:44:18PM +0900, Ian Lawrence Barwick wrote: > > > > + else if (child_relkind == > > > RELKIND_FOREIGN_TABLE && is_partitioned) > > > +

Re: Limit of WAL LSN value of type pg_lsn

2022-10-31 Thread Ashutosh Bapat
Hi Shubham, On Mon, Oct 31, 2022 at 11:04 AM Shubham Shingne wrote: > > Hi all, > > I am trying to determine optimal standby for automatic failover based on > pg_last_wal_recieve_lsn() value of all slaves. > > But what if max_wal_size is reached, is LSN value reset to zero. Per

Re: Commitfest documentation

2022-10-31 Thread Aleksander Alekseev
Hi Jehan-Guillaume, > In the commitfest application, I was wondering today what was the exact > meaning > and difference between open/closed status (is it only for the current > commitfest?) Closed means that the CF was in the past. It is archived now. Open means that new patches are accepted

Re: WIN32 pg_import_system_collations

2022-10-31 Thread Peter Eisentraut
On 12.07.22 21:32, Juan José Santamaría Flecha wrote: Please find attached a rebased version. I have split the patch into two parts trying to make it easier to review, one with the code changes and the other with the test. Other than that, there are minimal changes from the previous version

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-31 Thread Amit Kapila
On Fri, Oct 28, 2022 at 3:04 PM shiy.f...@fujitsu.com wrote: > > On Tue, Oct 25, 2022 2:56 PM Wang, Wei/王 威 wrote: > > I tried to write a draft patch to force streaming every change instead of > waiting until logical_decoding_work_mem is exceeded, which could help to test > streaming parallel.

Re: ResourceOwner refactoring

2022-10-31 Thread Aleksander Alekseev
Hi Heikki, > Rebased version attached. Given that Aleksander marked this as Ready for > Committer earlier, I'll add this to the next commitfest in that state, > and will commit in the next few days, barring any new objections. Thanks for resurrecting this patch. While taking a fresh look at the

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Dilip Kumar
On Mon, Oct 31, 2022 at 4:23 PM Dilip Kumar wrote: > > On Sun, Oct 30, 2022 at 11:32 PM Simon Riggs > wrote: > > > > On Fri, 28 Oct 2022 at 10:33, Simon Riggs > > wrote: > > > > > Thanks for the feedback, I will make all of those corrections in the > > > next version. > > > > New version

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Dilip Kumar
On Sun, Oct 30, 2022 at 11:32 PM Simon Riggs wrote: > > On Fri, 28 Oct 2022 at 10:33, Simon Riggs > wrote: > > > Thanks for the feedback, I will make all of those corrections in the > > next version. > > New version attached. I've rolled 002-004 into one patch, but can > split again as needed.

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Simon Riggs
On Mon, 31 Oct 2022 at 11:33, Dilip Kumar wrote: > What is the behavior if "nested_transactions" value is changed within > a transaction execution, suppose the value was on and we have created > a few levels of nested subtransactions and within the same transaction > I switched it to off or to

Re: Proposal to use JSON for Postgres Parser format

2022-10-31 Thread Matthias van de Meent
On Mon, 31 Oct 2022 at 13:46, Alexander Korotkov wrote: > On Fri, Oct 28, 2022 at 4:27 PM Andrew Dunstan wrote: >> On 2022-10-27 Th 19:38, Andres Freund wrote: >> > Hi, >> > >> > On 2022-09-19 22:29:15 -0400, Tom Lane wrote: >> >> Maybe a compromise could be found whereby we provide a conversion

Re: ResourceOwner refactoring

2022-10-31 Thread Heikki Linnakangas
On 12/01/2022 07:57, Julien Rouhaud wrote: On Fri, Nov 26, 2021 at 8:41 PM Aleksander Alekseev wrote: The patchset is in a good shape. I'm changing the status to "Ready for Committer". The 2nd patch doesn't apply anymore due to a conflict on resowner_private.h:

Re: resowner "cold start" overhead

2022-10-31 Thread Heikki Linnakangas
On 31/10/2022 04:28, Kyotaro Horiguchi wrote: At Sat, 29 Oct 2022 13:00:25 -0700, Andres Freund wrote in One way to reduce the size increase would be to use the space for initialarr to store variables we don't need while initialarr is used. E.g. itemsarr, maxitems, lastarr are candidates. But

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Pavel Borisov
I was working on optimizing the LWLock queue in a little different way and I also did a benchmarking of Andres' original patch from this thread. [1] The results are quite impressive, indeed. Please feel free to see the results and join the discussion in [1] if you want. Best regards, Pavel [1]

Re: Simplifying our Trap/Assert infrastructure

2022-10-31 Thread Tom Lane
Peter Eisentraut writes: > I don't think we need separate definitions for frontend and backend, > since the contained Assert() will take care of the difference. So the > attached would be simpler. WFM. regards, tom lane

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Simon Riggs
On Mon, 31 Oct 2022 at 12:22, Dilip Kumar wrote: > > On Mon, Oct 31, 2022 at 5:03 PM Dilip Kumar wrote: > > > > On Mon, Oct 31, 2022 at 4:23 PM Dilip Kumar wrote: > > > > > > On Sun, Oct 30, 2022 at 11:32 PM Simon Riggs > > > wrote: > > > > > > > > On Fri, 28 Oct 2022 at 10:33, Simon Riggs >

Re: warn if GUC set to an invalid shared library

2022-10-31 Thread Tom Lane
Justin Pryzby writes: > On Sun, Oct 30, 2022 at 04:12:33PM -0700, Maciek Sakrejda wrote: >> Also, for what it's worth, I think requiring the libraries to be in >> place before running ALTER SYSTEM does not really seem that onerous. I >> can't really think of use cases it precludes. > Right now,

Re: [PATCH] minor optimization for ineq_histogram_selectivity()

2022-10-31 Thread Frédéric Yhuel
On 10/24/22 17:26, Frédéric Yhuel wrote: Hello, When studying the weird planner issue reported here [1], I came up with the attached patch. It reduces the probability of calling get_actual_variable_range(). The patch applies to the master branch. How to test : CREATE TABLE foo (a

Re: Support logical replication of DDLs

2022-10-31 Thread vignesh C
On Thu, 27 Oct 2022 at 16:02, vignesh C wrote: > > On Thu, 27 Oct 2022 at 02:09, Zheng Li wrote: > > > > > Adding support for deparsing of CREATE/ALTER/DROP LANGUAGE for ddl > > > replication. > > > > Adding support for deparsing of: > > COMMENT > > ALTER DEFAULT PRIVILEGES > > CREATE/DROP

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Dilip Kumar
On Mon, Oct 31, 2022 at 5:03 PM Dilip Kumar wrote: > > On Mon, Oct 31, 2022 at 4:23 PM Dilip Kumar wrote: > > > > On Sun, Oct 30, 2022 at 11:32 PM Simon Riggs > > wrote: > > > > > > On Fri, 28 Oct 2022 at 10:33, Simon Riggs > > > wrote: > > > > > > > Thanks for the feedback, I will make all

Re: Adding doubly linked list type which stores the number of items in the list

2022-10-31 Thread Aleksander Alekseev
Hi hackers, > I will take another look at v3 tomorrow and probably mark it RfC. I very much like the patch. While on it: ``` +static inline bool +dclist_is_empty(dclist_head *head) +{ +Assert(dlist_is_empty(>dlist) == (head->count == 0)); +return (head->count == 0); +} ``` Should we

heapgettup refactoring

2022-10-31 Thread Melanie Plageman
Hi, Attached is a patchset to refactor heapgettup(), heapgettup_pagemode(), and heapgetpage(). heapgettup() and heapgettup_pagemode() have a lot of duplicated code, confusingly nested if statements, and unnecessary local variables. While working on a feature for the AIO/DIO patchset, I noticed

Re: Commitfest documentation

2022-10-31 Thread Jehan-Guillaume de Rorthais
Hi Aleksander, Thank you for your help! On Mon, 31 Oct 2022 16:51:23 +0300 Aleksander Alekseev wrote: [...] > > In the commitfest application, I was wondering today what was the exact > > meaning and difference between open/closed status (is it only for the > > current commitfest?) > >

Re: Improving btree performance through specializing by key shape, take 2

2022-10-31 Thread Matthias van de Meent
On Wed, 27 Jul 2022 at 13:34, Matthias van de Meent wrote: > > On Wed, 27 Jul 2022 at 09:35, Matthias van de Meent > wrote: > > > > On Mon, 4 Jul 2022 at 16:18, Matthias van de Meent > > wrote: > > > > > > On Sun, 5 Jun 2022 at 21:12, Matthias van de Meent > > > wrote: > > > > While working on

Re: Proposal to use JSON for Postgres Parser format

2022-10-31 Thread Michel Pelletier
On Mon, Oct 31, 2022 at 6:15 AM Matthias van de Meent < boekewurm+postg...@gmail.com> wrote: > On Mon, 31 Oct 2022 at 13:46, Alexander Korotkov > wrote: > > On Fri, Oct 28, 2022 at 4:27 PM Andrew Dunstan > wrote: > >> On 2022-10-27 Th 19:38, Andres Freund wrote: > >> > Hi, > >> > > >> > On

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Robert Haas
On Thu, Oct 27, 2022 at 12:59 PM Andres Freund wrote: > After pulling my > hair out for quite a while to try to understand that behaviour, I figured out > that it's just a side-effect of *removing* some other contention. I've seen this kind of pattern on multiple occasions. I don't know if they

Re: shared memory stats ideas

2022-10-31 Thread Peter Geoghegan
On Fri, Oct 21, 2022 at 2:26 AM Kyotaro Horiguchi wrote: > Stickier buffers for index pages seems to be related. I haven't see it > even get started, though. But this might be able be an additional > reason for starting it. Maybe, but FWIW I think that that will mostly just need to distinguish

Re: ResourceOwner refactoring

2022-10-31 Thread Aleksander Alekseev
Hi hackers, > > Rebased version attached. Given that Aleksander marked this as Ready for > > Committer earlier, I'll add this to the next commitfest in that state, > > and will commit in the next few days, barring any new objections. > > Thanks for resurrecting this patch. Additionally I decided

Lockless queue of waiters in LWLock

2022-10-31 Thread Pavel Borisov
Hi, hackers! When we take LWlock, we already use atomic CAS operation to atomically modify the lock state even in the presence of concurrent lock-takers. But if we can not take the lock immediately, we need to put the waiters on a waiting list, and currently, this operation is done not atomically

Re: hash_xlog_split_allocate_page: failed to acquire cleanup lock

2022-10-31 Thread Amit Kapila
On Tue, Oct 18, 2022 at 8:25 PM Robert Haas wrote: > > On Mon, Oct 17, 2022 at 4:30 PM Andres Freund wrote: > > On 2022-10-17 13:34:02 -0400, Robert Haas wrote: > > Maybe just nuking the IsBufferCleanupOK call is best, I don't know. I > honestly doubt that it matters very much what we pick here.

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Alexander Korotkov
Hi Andres, Thank you for your patch. The results are impressive. On Mon, Oct 31, 2022 at 2:10 PM Pavel Borisov wrote: > > I was working on optimizing the LWLock queue in a little different way > and I also did a benchmarking of Andres' original patch from this > thread. [1] > The results are

Re: Segfault on logical replication to partitioned table with foreign children

2022-10-31 Thread ilya . v . gladyshev
On Sun, 2022-10-30 at 09:39 -0400, Tom Lane wrote: > Dilip Kumar writes: > > Yes, this looks like a bug and your fix seems correct to me.  It > > would > > be nice to add a test case for this scenario. > > A test case doesn't seem that exciting to me.  If we were trying to > make it actually

Split index and table statistics into different types of stats

2022-10-31 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch proposal to split index and table statistics into different types of stats. This idea has been proposed by Andres in a couple of threads, see [1] and [2]. To sum up: We currently track index and table types of statistics in the same format (so

Re: Split index and table statistics into different types of stats

2022-10-31 Thread Drouvot, Bertrand
Hi, On 10/31/22 2:31 PM, Justin Pryzby wrote: I didn't looks closely, but there's a couple places where you wrote ";;", which looks unintentional. - PG_RETURN_TIMESTAMPTZ(tabentry->lastscan); + PG_RETURN_TIMESTAMPTZ(tabentry->lastscan);; Thanks for looking at it!

Re: hash_xlog_split_allocate_page: failed to acquire cleanup lock

2022-10-31 Thread Robert Haas
On Mon, Oct 31, 2022 at 7:27 AM Amit Kapila wrote: > Agreed, I think the important point to decide is what to do for > back-branches. We have the next minor release in a few days' time and > this is the last release for v10. I see the following options based on > the discussion here. > > a. Use

Re: Lockless queue of waiters in LWLock

2022-10-31 Thread Andres Freund
Hi, Thanks for working on this - I think it's something we need to improve. On 2022-10-31 14:38:23 +0400, Pavel Borisov wrote: > If the lock state contains references to the queue head and tail, we can > implement a lockless queue of waiters for the LWLock. Adding new items to > the queue head

Re: 16: Collation versioning and dependency helpers

2022-10-31 Thread Jeff Davis
On Sun, 2022-10-30 at 19:10 +1300, Thomas Munro wrote: > FWIW we did this (plus a lot more) in the per-index version tracking > feature reverted from 14. Thank you. I will catch up on that patch/thread. > >   0002: Enable pg_collation_actual_version() to work on the default > > Makes sense. >

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 17:17:03 -0700, Zhihong Yu wrote: > On Mon, Oct 31, 2022 at 4:51 PM Andres Freund wrote: > > > Hi, > > > > On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > > > BTW, I've seen a sporadic crash (SEGV) with the patch in bg writer > > > with the same set up [1], I'm not

User functions for building SCRAM secrets

2022-10-31 Thread Jonathan S. Katz
Hi, We currently do not provide any SQL functions for generating SCRAM secrets, whereas we have this support for other passwords types (plaintext and md5 via `md5(password || username)`). If a user wants to build a SCRAM secret via SQL, they have to implement our SCRAM hashing funcs on their

Re: RLS + XPATH

2022-10-31 Thread Joe Conway
On 10/31/22 13:37, Joshua Drake wrote: Team, While on the road in Iowa visiting covered bridges I met up with an amazing individual named Brent. Brent, is with a small organization named: Darpa. They are using PostgreSQL + RLS + XPATH but unfortunately the performance has been less than

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2022-10-31 Thread Jacob Champion
On Tue, Oct 25, 2022 at 1:20 PM Jacob Champion wrote: > I wanted to get feedback on the approach before wordsmithing too > much. I've added this to tomorrow's CF [1]. Thomas, if you get (or already have) a PG community username, I can add you as an author. Thanks, --Jacob [1]

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > BTW, I've seen a sporadic crash (SEGV) with the patch in bg writer > with the same set up [1], I'm not sure if it's really because of the > patch. I'm unable to reproduce it now and unfortunately I didn't > capture further details when

Re: Split index and table statistics into different types of stats

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 14:14:15 +0100, Drouvot, Bertrand wrote: > Please find attached a patch proposal to split index and table statistics > into different types of stats. > > This idea has been proposed by Andres in a couple of threads, see [1] and > [2]. Thanks for working on this! > diff

Re: Simplifying our Trap/Assert infrastructure

2022-10-31 Thread Michael Paquier
On Mon, Oct 31, 2022 at 09:14:10AM -0400, Tom Lane wrote: > Peter Eisentraut writes: >> I don't think we need separate definitions for frontend and backend, >> since the contained Assert() will take care of the difference. So the >> attached would be simpler. > > WFM. Thanks, fine by me. --

Re: User functions for building SCRAM secrets

2022-10-31 Thread Dagfinn Ilmari Mannsåker
"Jonathan S. Katz" writes: > Attached is a (draft) patch that adds a function called > "scram_build_secret_sha256" that can take 3 arguments: This seems like a reasonable piece of functionality, I just have one comment on the implementation. > * password (text) - a plaintext password > * salt

Re: Prefetch the next tuple's memory during seqscans

2022-10-31 Thread David Rowley
On Tue, 1 Nov 2022 at 03:12, Aleksander Alekseev wrote: > I wonder if we can be sure and/or check that there is no performance > degradation under different loads and different platforms... Different platforms would be good. Certainly, 1 platform isn't a good enough indication that this is going

Reusing return value from planner_rt_fetch

2022-10-31 Thread Zhihong Yu
Hi, I was reading examine_variable in src/backend/utils/adt/selfuncs.c It seems we already have the rte coming out of the loop which starts on line 5181. Here is a patch which reuses the return value from `planner_rt_fetch`. Please take a look. Thanks parent-rel-rte.patch Description: Binary

Re: [PATCHES] Post-special page storage TDE support

2022-10-31 Thread Matthias van de Meent
On Sat, 29 Oct 2022 at 00:25, David Christensen wrote: > > Hi Matthias, > > > Did you read the related thread with related discussion from last June, > > "Re: better page-level checksums" [0]? In that I argued that space at the > > end of a page is already allocated for the AM, and that

Re: Segfault on logical replication to partitioned table with foreign children

2022-10-31 Thread Ilya Gladyshev
On Mon, 2022-10-31 at 03:20 +, shiy.f...@fujitsu.com wrote: > On Sun, Oct 30, 2022 9:39 PM Tom Lane wrote: > > > > What I'm wondering about is whether we can refactor this code > > to avoid so many usually-useless catalog lookups.  Pulling the > > namespace name, in particular, is expensive

Re: resowner "cold start" overhead

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 11:05:32 +0100, Heikki Linnakangas wrote: > What do you think of my ResourceOwner refactoring patches [1]? Reminded by > this, I rebased and added it to the upcoming commitfest again. > With that patch, all resources are stored in the same array and hash. The > array is part of

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Zhihong Yu
On Mon, Oct 31, 2022 at 4:51 PM Andres Freund wrote: > Hi, > > On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > > BTW, I've seen a sporadic crash (SEGV) with the patch in bg writer > > with the same set up [1], I'm not sure if it's really because of the > > patch. I'm unable to reproduce

Re: Segfault on logical replication to partitioned table with foreign children

2022-10-31 Thread Ilya Gladyshev
On Sun, 2022-10-30 at 16:52 +0100, Alvaro Herrera wrote: > On 2022-Oct-28, ilya.v.gladys...@gmail.com wrote: > > > This will cause a segfault or raise an assert, because inserting > > into > > foreign tables via logical replication is not possible. The > > solution I > > propose is to add

Re: heapgettup refactoring

2022-10-31 Thread Justin Pryzby
FYI: [18:51:54.707] ../src/backend/access/heap/heapam.c(720): warning C4098: 'heapgettup': 'void' function returning a value [18:51:54.707] ../src/backend/access/heap/heapam.c(850): warning C4098: 'heapgettup_pagemode': 'void' function returning a value For some reason, MSVC is the only one to

Re: User functions for building SCRAM secrets

2022-10-31 Thread Jonathan S. Katz
On 10/31/22 6:05 PM, Dagfinn Ilmari Mannsåker wrote: * password (text) - a plaintext password * salt (text) - a base64 encoded salt […] + /* +* determine if this a valid base64 encoded string +* TODO: look into refactoring the SCRAM decode code in libpq/auth-scram.c +

Re: ResourceOwner refactoring

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 10:51:36 +0100, Heikki Linnakangas wrote: > These are functions where quite a lot of things happen between the > ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that > there are no unrelated ResourceOwnerRemember() calls in the code in > between, otherwise

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2022-10-31 Thread Bharath Rupireddy
On Mon, Oct 31, 2022 at 11:50 AM Bharath Rupireddy wrote: > > On Mon, Oct 31, 2022 at 5:01 AM Michael Paquier wrote: > > > > On Sun, Oct 30, 2022 at 03:44:32PM +0100, Alvaro Herrera wrote: > > > So I'm kinda proposing that we only do the forward struct initialization > > > dance when it really

Re: Commit fest 2022-11

2022-10-31 Thread Michael Paquier
On Tue, Nov 01, 2022 at 10:01:05AM +0900, Ian Lawrence Barwick wrote: > This is on my bucket list of things to do some day, so I guess now is as bad a > time as any :). Caveat is that this will have to be a personal > free-time project, > so would be good if someone else is around as well. Don't

Re: Check return value of pclose() correctly

2022-10-31 Thread Michael Paquier
On Mon, Oct 31, 2022 at 09:12:53AM +0100, Peter Eisentraut wrote: > I noticed that some (not all) callers didn't check the return value of > pclose() or ClosePipeStream() correctly. Either they didn't check it at all > or they treated it like the return of fclose(). Here is a patch with fixes. >

Re: [patch] Have psql's \d+ indicate foreign partitions

2022-10-31 Thread Michael Paquier
On Mon, Oct 24, 2022 at 09:44:18PM +0900, Ian Lawrence Barwick wrote: > Recently I have been working a lot with partitioned tables which contain a mix > of local and foreign partitions, and find it would be very useful to be able > to > easily obtain an overview of which partitions are foreign

Re: Commit fest 2022-11

2022-10-31 Thread Ian Lawrence Barwick
2022年10月31日(月) 14:42 Michael Paquier : > > Hi all, > > As per the world clock, the next commit fest will begin in 30 hours > (11/1 0:00 AoE time). I may have missed something, but it looks like > we have no CFM for this one yet. ** tumbleweed ** > Opinions, thoughts or volunteers? This is on

Re: hash_xlog_split_allocate_page: failed to acquire cleanup lock

2022-10-31 Thread Amit Kapila
On Mon, Oct 31, 2022 at 10:40 PM Robert Haas wrote: > > On Mon, Oct 31, 2022 at 7:27 AM Amit Kapila wrote: > > Agreed, I think the important point to decide is what to do for > > back-branches. We have the next minor release in a few days' time and > > this is the last release for v10. I see the

Re: Commit fest 2022-11

2022-10-31 Thread Michael Paquier
On Tue, Nov 01, 2022 at 09:56:47AM +0530, jian he wrote: > I am free. I can help. Commit fest managers are usually people who have a few years of experience behind the community process. There are plenty of patches to review, so feel free to pick up a few things and help moving these, of course!

Re: heapgettup refactoring

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 14:37:39 -0400, Melanie Plageman wrote: > and heapgetpage(). heapgettup() and heapgettup_pagemode() have a lot of > duplicated code, confusingly nested if statements, and unnecessary local > variables. While working on a feature for the AIO/DIO patchset, I > noticed that it was

Re: Reusing return value from planner_rt_fetch

2022-10-31 Thread Tom Lane
Zhihong Yu writes: > I was reading examine_variable in src/backend/utils/adt/selfuncs.c > It seems we already have the rte coming out of the loop which starts on > line 5181. > Here is a patch which reuses the return value from `planner_rt_fetch`. planner_rt_fetch is not so expensive that we

Re: Commit fest 2022-11

2022-10-31 Thread jian he
On Tue, Nov 1, 2022 at 6:31 AM Ian Lawrence Barwick wrote: > 2022年10月31日(月) 14:42 Michael Paquier : > > > > Hi all, > > > > As per the world clock, the next commit fest will begin in 30 hours > > (11/1 0:00 AoE time). I may have missed something, but it looks like > > we have no CFM for this

Re: Check return value of pclose() correctly

2022-10-31 Thread Tom Lane
Michael Paquier writes: > On Mon, Oct 31, 2022 at 09:12:53AM +0100, Peter Eisentraut wrote: >> (A failure to run the command issued by popen() is usually reported via the >> pclose() status, so while you can often get away with not checking fclose() >> or close(), checking pclose() is more often

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Zhihong Yu
On Mon, Oct 31, 2022 at 5:19 PM Andres Freund wrote: > Hi, > > On 2022-10-31 17:17:03 -0700, Zhihong Yu wrote: > > On Mon, Oct 31, 2022 at 4:51 PM Andres Freund > wrote: > > > > > Hi, > > > > > > On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > > > > BTW, I've seen a sporadic crash

Re: User functions for building SCRAM secrets

2022-10-31 Thread Michael Paquier
On Mon, Oct 31, 2022 at 04:27:08PM -0400, Jonathan S. Katz wrote: > 1. password only -- this defers to the PG defaults for SCRAM > 2. password + salt -- this is useful for the password history / dictionary > case to allow for a predictable way to check a password. Well, one could pass a salt

Re: Lock on ShmemVariableCache fields?

2022-10-31 Thread Michael Paquier
On Mon, Oct 31, 2022 at 03:14:54PM +0800, Japin Li wrote: > For example, since SetCommitTsLimit() is only used in BootStrapXLog() and > StartupXLOG(), we can safely remove the code of acquiring/releasing lock? Logically yes, I guess that you could go without the LWLock acquired in this routine at

Re: Simplifying our Trap/Assert infrastructure

2022-10-31 Thread Peter Eisentraut
On 31.10.22 01:04, Michael Paquier wrote: On Fri, Oct 28, 2022 at 09:36:23AM +0200, Peter Eisentraut wrote: Would there be a use for that? It's currently only used in the atomics code. Yep, but they would not trigger when using atomics in the frontend code. We don't have any use for that in

Error for row-level triggers with transition tables on partitioned tables

2022-10-31 Thread Etsuro Fujita
While working on something else, I noticed $SUBJECT: we do not currently allow row-level triggers on partitioned tables to have transition tables like this: create table parted_trig (a int) partition by list (a); CREATE TABLE create function trigger_nothing() returns trigger language plpgsql as

Re: Adding doubly linked list type which stores the number of items in the list

2022-10-31 Thread Bharath Rupireddy
On Mon, Oct 31, 2022 at 6:58 PM Aleksander Alekseev wrote: > > Hi hackers, > > > I will take another look at v3 tomorrow and probably mark it RfC. > > I very much like the patch. While on it: > > ``` > +static inline bool > +dclist_is_empty(dclist_head *head) > +{ > +

Re: Commitfest documentation

2022-10-31 Thread Jacob Champion
On Mon, Oct 31, 2022 at 8:18 AM Jehan-Guillaume de Rorthais wrote: > However, I'm not strictly sure who is responsible to set these statuses. The > reviewer? The author? The commiter? The CF manager? I bet on the reviewer, but > it seems weird a random reviewer can reject a patch on its own