Re: OpenSSL 3.0.0 compatibility

2021-03-11 Thread Peter Eisentraut
On 11.03.21 11:41, Daniel Gustafsson wrote: .. and apply the padding changes as proposed in a patch upthread like this (these work for all OpenSSL versions I've tested, and I'm rather more puzzled as to why we got away with not having them in the past): Yes, before proceeding with this, we

Re: Different compression methods for FPI

2021-03-11 Thread Justin Pryzby
On Sat, Mar 06, 2021 at 12:29:14PM +0500, Andrey Borodin wrote: > > 1 марта 2021 г., в 10:03, Justin Pryzby написал(а): > > Justin, Michael, thanks for comments! > > As far as I understood TODO list for the patch looks as follows: Your patch can be simplified some, and then the only ifdef are

RE: Enhance traceability of wal_level changes for backup management

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > As a backup software author, I don't see this feature as very useful. > > The problem is that there are lots of ways for WAL to go missing so > monitoring the WAL archive for gaps is essential and this feature would > not replace that requirement. The only extra information

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-11 Thread Greg Nancarrow
On Fri, Mar 12, 2021 at 5:00 AM Tom Lane wrote: > > > The problem is that target_rel_trigger_max_parallel_hazard and its caller > think they can use a relcache TriggerDesc field across other cache > accesses, which they can't because the relcache doesn't guarantee that > that won't move. > > One

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-11 Thread Amit Langote
On Fri, Mar 12, 2021 at 6:10 AM Justin Pryzby wrote: > Note also this CF entry > https://commitfest.postgresql.org/32/2987/ > | Allow setting parallel_workers on partitioned tables +/* + * PartitionedOptions + * Contents of rd_options for partitioned tables + */ +typedef struct

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-11 Thread Thomas Munro
On Thu, Mar 11, 2021 at 7:34 PM Michael Paquier wrote: > On Thu, Mar 11, 2021 at 04:37:39PM +1300, Thomas Munro wrote: > > Michael, when you said "That's pretty hack-ish, still efficient" in > > reference to this code: > > > >> - if (IsUnderPostmaster && !PostmasterIsAlive()) > >> + if

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread k.jami...@fujitsu.com
> From: Tsunakawa, Takayuki/綱川 貴之 > From: Kyotaro Horiguchi > > About the patch, it would be better to change the type of > > BUF_DROP_FULL_SCAN_THRESHOLD to uint64, even though the current > value > > doesn't harm. > > OK, attached, to be prepared for the distant future when NBuffers becomes >

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > About the patch, it would be better to change the type of > BUF_DROP_FULL_SCAN_THRESHOLD to uint64, even though the current > value > doesn't harm. OK, attached, to be prepared for the distant future when NBuffers becomes 64-bit. Regards Takayuki Tsunakawa

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Dilip Kumar
On Fri, Mar 12, 2021 at 11:55 AM Justin Pryzby wrote: > > It worked everywhere (but everytime someone mail this thread, it queues a > rebuild). > http://cfbot.cputube.org/dilip-kumar.html Okay > HAVE_LIBLZ4 was being set by AC_CHECK_LIB(), which is no longer used in favour > of pkgconfig. >

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Justin Pryzby
On Fri, Mar 12, 2021 at 11:24:57AM +0530, Dilip Kumar wrote: > On Fri, Mar 12, 2021 at 10:45 AM Justin Pryzby wrote: > > > > On Thu, Mar 11, 2021 at 12:25:26PM -0600, Justin Pryzby wrote: > > > On Wed, Mar 10, 2021 at 08:28:58PM -0600, Justin Pryzby wrote: > > > > This includes a patch to use

Re: shared-memory based stats collector

2021-03-11 Thread Fujii Masao
On 2021/03/12 13:49, Kyotaro Horiguchi wrote: At Fri, 12 Mar 2021 10:38:00 +0900 (JST), Kyotaro Horiguchi wrote in At Fri, 12 Mar 2021 10:03:31 +0900, Fujii Masao wrote in I moved archiver from the current location to next to "walsenders" that is to be terminated along with archiver.

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread Kyotaro Horiguchi
At Fri, 12 Mar 2021 05:26:02 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Thomas Munro > > > uint64 > > > > +1 > > Thank you, the patch is attached (we tend to forget how large our world is... > 64-bit) We're sorry to cause you trouble. BUF_DROP_FULL_SCAN_THRESHOLD cannot be

Re: [PATCH] Hooks at XactCommand level

2021-03-11 Thread Julien Rouhaud
Hi, On Tue, Dec 08, 2020 at 11:15:12AM +0100, Gilles Darold wrote: > > Based on a PoC reported in a previous thread [1] I'd like to propose new > hooks around transaction commands. The objective of this patch is to > allow PostgreSQL extension to act at start and end (including abort) of > a SQL

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Dilip Kumar
On Fri, Mar 12, 2021 at 10:45 AM Justin Pryzby wrote: > > On Thu, Mar 11, 2021 at 12:25:26PM -0600, Justin Pryzby wrote: > > On Wed, Mar 10, 2021 at 08:28:58PM -0600, Justin Pryzby wrote: > > > This includes a patch to use pkgconfig, in an attempt to build on mac, > > > which > > > currently

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > Could you rebase again and send an updated patch ? > I could do it if you want. Rebased and attached. Fortunately, there was no rebase conflict this time. make check passed for PG core and postgres_fdw. Regards Takayuki Tsunakawa

Re: shared-memory based stats collector

2021-03-11 Thread Kyotaro Horiguchi
At Thu, 11 Mar 2021 19:22:57 -0800, Andres Freund wrote in > Hi, > > On 2021-03-10 20:26:56 -0800, Andres Freund wrote: > > > +static void > > > +attach_shared_stats(void) > > > +{ > > > + MemoryContext oldcontext; > > > > > + /* > > > + * The first attacher backend may still reading the

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > > uint64 > > +1 Thank you, the patch is attached (we tend to forget how large our world is... 64-bit) We're sorry to cause you trouble. Regards Takayuki Tsunakawa v1-0001-Fix-overflow-when-counting-the-number-of-buffers-.patch Description:

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-03-11 Thread Fujii Masao
On 2021/03/11 23:33, Fujii Masao wrote: On 2021/03/11 21:29, Masahiro Ikeda wrote: On 2021-03-11 11:52, Fujii Masao wrote: On 2021/03/11 9:38, Masahiro Ikeda wrote: On 2021-03-10 17:08, Fujii Masao wrote: On 2021/03/10 14:11, Masahiro Ikeda wrote: On 2021-03-09 17:51, Fujii Masao

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-11 Thread vignesh C
On Thu, Mar 11, 2021 at 7:20 AM Peter Smith wrote: > > On Thu, Mar 11, 2021 at 12:46 PM Peter Smith wrote: > > > > Please find attached the latest patch set v57* > > > > Differences from v56* are: > > > > * Rebased to HEAD @ today > > > > * Addresses the following feedback issues: > > > > (24)

Re: shared-memory based stats collector

2021-03-11 Thread Kyotaro Horiguchi
At Fri, 12 Mar 2021 10:38:00 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 12 Mar 2021 10:03:31 +0900, Fujii Masao > wrote in > > > I moved archiver from the current location to next to "walsenders" > > > that is to be terminated along with archiver. > > > The attached the only 0003 of

Re: Performance degradation of REFRESH MATERIALIZED VIEW

2021-03-11 Thread Masahiko Sawada
On Fri, Mar 12, 2021 at 3:13 AM Andres Freund wrote: > > Hi, > > On 2021-03-11 17:44:37 +0900, Masahiko Sawada wrote: > > The execution time of REFRESH MATERIALIZED VIEW are: > > > > w/ HEAP_INSERT_FROZEN flag : 42 sec > > w/o HEAP_INSERT_FROZEN flag : 33 sec > > > > After investigation, I found

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > On Fri, Mar 12, 2021 at 5:20 PM Amit Kapila wrote: > > uint64 > > +1 +1 I'll send a patch later. Regards Takayuki Tsunakawa

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread Thomas Munro
On Fri, Mar 12, 2021 at 5:20 PM Amit Kapila wrote: > uint64 +1

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Dilip Kumar
On Fri, Mar 12, 2021 at 2:54 AM Justin Pryzby wrote: > > On Thu, Mar 11, 2021 at 10:07:30AM +0530, Dilip Kumar wrote: > > On Thu, Mar 11, 2021 at 8:50 AM Justin Pryzby wrote: > > > > > > Looking at v23-0002-alter-table-set-compression, ATRewriteTable() was > > > calling > > >

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread Amit Kapila
On Fri, Mar 12, 2021 at 4:58 AM Thomas Munro wrote: > > While rebasing CF #2933 (which drops the _cached stuff and makes this > optimisation always available, woo), I happened to notice that we're > summing the size of many relations and forks into a variable > nBlocksToInvalidate of type

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Dilip Kumar
On Fri, Mar 12, 2021 at 9:03 AM Justin Pryzby wrote: > > On Fri, Mar 12, 2021 at 08:38:41AM +0530, Dilip Kumar wrote: > > On Thu, Mar 11, 2021 at 11:55 PM Justin Pryzby wrote: > > > On Wed, Mar 10, 2021 at 08:28:58PM -0600, Justin Pryzby wrote: > > > > This includes a patch to use pkgconfig, in

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Dilip Kumar
On Fri, Mar 12, 2021 at 8:54 AM Justin Pryzby wrote: > > +#elif LZ4_VERSION_NUMBER < 10803 > + return lz4_cmdecompress(value); > +#else > > It occurred to me that this should actually compare the runtime version with > LZ4_versionNumber(). That way, a library upgrade can enable the slice >

RE: Parallel INSERT (INTO ... SELECT ...)

2021-03-11 Thread houzj.f...@fujitsu.com
> On Thu, Mar 11, 2021 at 01:01:42PM +, houzj.f...@fujitsu.com wrote: > > > I guess to have the finer granularity we'd have to go with > > > enable_parallel_insert, which then would mean possibly having to > > > later add enable_parallel_update, should parallel update have > > > similar

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-03-11 Thread Fujii Masao
On 2021/03/11 21:29, Masahiro Ikeda wrote: On 2021-03-11 11:52, Fujii Masao wrote: On 2021/03/11 9:38, Masahiro Ikeda wrote: On 2021-03-10 17:08, Fujii Masao wrote: On 2021/03/10 14:11, Masahiro Ikeda wrote: On 2021-03-09 17:51, Fujii Masao wrote: On 2021/03/05 8:38, Masahiro Ikeda

Re: Huge memory consumption on partitioned table with FKs

2021-03-11 Thread Tatsuro Yamada
On 2021/03/12 2:44, Tom Lane wrote: I wrote: Now, maybe it's a coincidence that husky failed on a partitioned-foreign-key test right after this patch went in, but I bet not. Since husky runs CLOBBER_CACHE_ALWAYS, it looks to me like we've overlooked some cache-reset scenario or other. After

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Justin Pryzby
On Fri, Mar 12, 2021 at 08:38:41AM +0530, Dilip Kumar wrote: > On Thu, Mar 11, 2021 at 11:55 PM Justin Pryzby wrote: > > On Wed, Mar 10, 2021 at 08:28:58PM -0600, Justin Pryzby wrote: > > > This includes a patch to use pkgconfig, in an attempt to build on mac, > > > which > > > currently fails

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Justin Pryzby
On Mon, Mar 01, 2021 at 08:53:09PM +0530, Dilip Kumar wrote: > On Mon, Mar 1, 2021 at 5:36 PM Dilip Kumar wrote: > > On Mon, Mar 1, 2021 at 11:06 AM Justin Pryzby wrote: > > > Thanks. It seems like that explains it. > > > I think if that's a problem with recent versions, then you'll have to > >

Re: shared-memory based stats collector

2021-03-11 Thread Andres Freund
Hi, On 2021-03-10 20:26:56 -0800, Andres Freund wrote: > > +static void > > +attach_shared_stats(void) > > +{ > > + MemoryContext oldcontext; > > > + /* > > +* The first attacher backend may still reading the stats file, or the > > +* last detacher may writing it. Wait for the work

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Dilip Kumar
On Thu, Mar 11, 2021 at 11:55 PM Justin Pryzby wrote: > > On Wed, Mar 10, 2021 at 08:28:58PM -0600, Justin Pryzby wrote: > > This includes a patch to use pkgconfig, in an attempt to build on mac, which > > currently fails like: > > > >

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-11 Thread David Rowley
On Fri, 12 Mar 2021 at 14:59, Tom Lane wrote: > > David Rowley writes: > > The 0001 patch adds a has_volatile bool field to RestrictInfo and sets > > it when building the RestrictInfo. > > I'm -1 on doing it exactly that way, because you're expending > the cost of those lookups without certainty

Re: Allow batched insert during cross-partition updates

2021-03-11 Thread Amit Langote
On Thu, Mar 11, 2021 at 8:36 PM wrote: > On Thursday, March 11, 2021 9:42 AM, Amit Langote > wrote: > > On Wed, Mar 10, 2021 at 9:30 PM Georgios gkokola...@protonmail.com wrote: > > > > > I continued looking a bit at the patch, yet I am either failing to see > > > fix or I am > > > looking at

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-11 Thread Tom Lane
David Rowley writes: > On Tue, 23 Feb 2021 at 18:43, Tom Lane wrote: >> I doubt it's that bad. We could cache such info in RestrictInfo >> for quals, or PathTarget for tlists, without much new notational >> overhead. That doesn't cover everything the planner deals with >> of course, but it

Re: Procedures versus the "fastpath" API

2021-03-11 Thread Noah Misch
On Wed, Mar 10, 2021 at 10:03:24AM +0100, Laurenz Albe wrote: > On Tue, 2021-03-09 at 14:15 -0500, Tom Lane wrote: > > As for procedures, I'm of the opinion that we should just reject those > > too, but some other security team members were not happy with that > > idea. Conceivably we could

Re: shared-memory based stats collector

2021-03-11 Thread Kyotaro Horiguchi
At Fri, 12 Mar 2021 10:03:31 +0900, Fujii Masao wrote in > > I moved archiver from the current location to next to "walsenders" > > that is to be terminated along with archiver. > > The attached the only 0003 of the new version based on the last one > > from Fujii-san. > > Thanks for updating

Re: shared-memory based stats collector

2021-03-11 Thread Kyotaro Horiguchi
At Fri, 12 Mar 2021 10:07:10 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 12 Mar 2021 09:23:12 +0900 (JST), Kyotaro Horiguchi > wrote in > > The attached the only 0003 of the new version based on the last one > > from Fujii-san. > > Please wait a moment. Something might be wrong. It

Re: shared-memory based stats collector

2021-03-11 Thread Kyotaro Horiguchi
At Fri, 12 Mar 2021 09:23:12 +0900 (JST), Kyotaro Horiguchi wrote in > The attached the only 0003 of the new version based on the last one > from Fujii-san. Please wait a moment. Something might be wrong. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: New IndexAM API controlling index vacuum strategies

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 8:31 AM Robert Haas wrote: > I agree, but all you need is one long-lived tuple toward the end of > the array and you're stuck never being able to truncate it. It seems > like a worthwhile improvement, but whether it actually helps will be > workload-dependant. When it

Re: shared-memory based stats collector

2021-03-11 Thread Fujii Masao
On 2021/03/12 9:23, Kyotaro Horiguchi wrote: At Thu, 11 Mar 2021 15:33:52 +0900, Fujii Masao wrote in On 2021/03/11 13:42, Kyotaro Horiguchi wrote: At Wed, 10 Mar 2021 19:21:00 -0800, Andres Freund wrote in Hi, Two minor nits: Thanks for the comments! On 2021-03-10 21:47:51 +0900,

Re: Is Recovery actually paused?

2021-03-11 Thread Dilip Kumar
On Fri, 12 Mar 2021 at 2:04 AM, Robert Haas wrote: > On Mon, Mar 1, 2021 at 12:08 AM Dilip Kumar wrote: > > > > > I'll wait for a day before marking this RfC in case anyone have > > > > > further comments. > > > > > > > > Okay. > > > > > > Hearing nothing, done that. > > > > Thanks. > >

Re: compression libraries and CF bot

2021-03-11 Thread Thomas Munro
On Sun, Feb 21, 2021 at 4:30 AM Justin Pryzby wrote: > Dilip's TOAST patch is passing on linux and bsd --with-lz4, so I think it's > desirable to install on mac now. Justin figured this out, so now this patch is using lz4 and passing on Linux, FreeBSD and macOS. > libzstd would be desirable for

Re: [PATCH] Covering SPGiST index

2021-03-11 Thread Tom Lane
I wrote: > I spent a little time looking at this, and realized something that may > or may not be a serious problem. This form of the patch supposes that > it can use the usual tuple form/deform logic for all columns of a leaf > tuple including the key column. However, that does not square with

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-11 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 3/12/21 1:11 AM, Stephen Frost wrote: > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > >> On 3/8/21 8:42 PM, Stephen Frost wrote: > >>> * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 2/10/21 11:10

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-11 Thread David Rowley
On Tue, 23 Feb 2021 at 14:22, Andy Fan wrote: > > On Mon, Feb 22, 2021 at 9:21 AM Justin Pryzby wrote: >> - Maybe this should be integrated into nestloop rather than being a separate >>plan node. That means that it could be dynamically enabled during >>execution, maybe after a few

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-11 Thread David Rowley
Thanks for these suggestions. On Mon, 22 Feb 2021 at 14:21, Justin Pryzby wrote: > > On Tue, Feb 16, 2021 at 11:15:51PM +1300, David Rowley wrote: > > To summarise here, the planner performance gets a fair bit worse with > > the patched code. With master, summing the average planning time over >

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-11 Thread Tomas Vondra
On 3/12/21 1:11 AM, Stephen Frost wrote: > Greetings, > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: >> On 3/8/21 8:42 PM, Stephen Frost wrote: >>> * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: On 2/10/21 11:10 PM, Stephen Frost wrote: > * Heikki Linnakangas

Re: shared-memory based stats collector

2021-03-11 Thread Kyotaro Horiguchi
At Thu, 11 Mar 2021 15:33:52 +0900, Fujii Masao wrote in > > > On 2021/03/11 13:42, Kyotaro Horiguchi wrote: > > At Wed, 10 Mar 2021 19:21:00 -0800, Andres Freund > > wrote in > >> Hi, > >> > >> Two minor nits: > > Thanks for the comments! > > > >> On 2021-03-10 21:47:51 +0900, Fujii Masao

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-11 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 3/8/21 8:42 PM, Stephen Frost wrote: > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > >> On 2/10/21 11:10 PM, Stephen Frost wrote: > >>> * Heikki Linnakangas (hlinn...@iki.fi) wrote: > On 05/02/2021 23:22,

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread Thomas Munro
While rebasing CF #2933 (which drops the _cached stuff and makes this optimisation always available, woo), I happened to notice that we're summing the size of many relations and forks into a variable nBlocksToInvalidate of type BlockNumber. That could overflow.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-11 Thread Greg Nancarrow
On Fri, Mar 12, 2021 at 5:00 AM Tom Lane wrote: > > The buildfarm has revealed that this patch doesn't work under > CLOBBER_CACHE_ALWAYS: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=husky=2021-03-10%2021%3A09%3A32 > > I initially thought that that was a problem with c3ffe34863,

Re: OpenSSL 3.0.0 compatibility

2021-03-11 Thread Daniel Gustafsson
> On 12 Mar 2021, at 00:04, Peter Eisentraut > wrote: > > On 11.03.21 11:41, Daniel Gustafsson wrote: >> Then there are a few where we get padding back where we really should have >> ended up with the "Cipher cannot be initialized" error since DES is in the >> legacy provider: >> select

Re: OpenSSL 3.0.0 compatibility

2021-03-11 Thread Peter Eisentraut
On 11.03.21 11:41, Daniel Gustafsson wrote: Then there are a few where we get padding back where we really should have ended up with the "Cipher cannot be initialized" error since DES is in the legacy provider: select decrypt_iv(decode('50735067b073bb93', 'hex'), '0123456', 'abcd', 'des'); -

Re: Self-join optimisation

2021-03-11 Thread Tomas Vondra
On 3/11/21 3:39 PM, Hywel Carver wrote: > Great! It looks like it's been in commitfest status for a few years. Is > there anything someone like me (outside the pgsql-hackers community) can > do to help it get reviewed this time around? > Well, you could do a review, or at least test it with the

Re: [PATCH] Covering SPGiST index

2021-03-11 Thread Tom Lane
David Steele writes: > Tom, do the changes as enumerated in [1] look like they are going in the > right direction? I spent a little time looking at this, and realized something that may or may not be a serious problem. This form of the patch supposes that it can use the usual tuple form/deform

Re: cursor sensitivity misunderstanding

2021-03-11 Thread Peter Eisentraut
On 09.03.21 00:22, David G. Johnston wrote: I came up with the attached patch to sort this out a bit.  It does not change any cursor behavior.  But the documentation now uses the terms more correctly and explains the differences between SQL and the PostgreSQL implementation

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
On Thu, Mar 11, 2021 at 11:09 AM Robert Haas wrote: > An alternate possibility would be to say that there should only ever > be EITHER a bare command-line argument OR options that require > querying for a list of databases OR neither BUT NOT both. Then it's > simple: > > 0. If you have both

Re: [HACKERS] Custom compression methods

2021-03-11 Thread Justin Pryzby
On Thu, Mar 11, 2021 at 10:07:30AM +0530, Dilip Kumar wrote: > On Thu, Mar 11, 2021 at 8:50 AM Justin Pryzby wrote: > > > > Looking at v23-0002-alter-table-set-compression, ATRewriteTable() was > > calling > > CompareCompressionMethodAndDecompress(). > > While changing the compression method

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-11 Thread Justin Pryzby
On Thu, Mar 11, 2021 at 01:01:42PM +, houzj.f...@fujitsu.com wrote: > > I guess to have the finer granularity we'd have to go with > > enable_parallel_insert, > > which then would mean possibly having to later add enable_parallel_update, > > should parallel update have similar potential

Re: Is Recovery actually paused?

2021-03-11 Thread Robert Haas
On Mon, Mar 1, 2021 at 12:08 AM Dilip Kumar wrote: > > > > I'll wait for a day before marking this RfC in case anyone have > > > > further comments. > > > > > > Okay. > > > > Hearing nothing, done that. > > Thanks. Committed with minor cosmetic changes. -- Robert Haas EDB:

Re: documentation fix for SET ROLE

2021-03-11 Thread David G. Johnston
On Thursday, March 11, 2021, Bossart, Nathan wrote: > Thanks for reviewing. > > On 3/11/21, 6:59 AM, "Laurenz Albe" wrote: > > I have had a look at the patch, and while I agree that this should > > be documented, I am not happy with the patch as it is. > > > > I think we should *not* document

Re: libpq debug log

2021-03-11 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Mar-10, Tom Lane wrote: >> After studying this further, I think we should apply the attached >> patch to remove that responsibility from pqParseInput3's subroutines. >> This will allow a single trace call near the bottom of pqParseInput3 >> to handle all cases

Re: pgbench - add pseudo-random permutation function

2021-03-11 Thread Dean Rasheed
On Thu, 11 Mar 2021 at 19:06, David Bowen wrote: > > The algorithm for generating a random permutation with a uniform distribution > across all permutations is easy: > for (i=0; iswap a[n-i] with a[rand(n-i+1)] > } > > where 0 <= rand(x) < x and a[i] is initially i (see Knuth, Section 3.4.2

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
On Wed, Mar 10, 2021 at 11:02 PM Mark Dilger wrote: > [ new patches ] Seems like this is mostly ready to commit now, modulo exactly what to do about the maintenance DB stuff, and whether to move it to src/bin. Since neither of those affects 0001, I went ahead and committed that part. -- Robert

Re: New IndexAM API controlling index vacuum strategies

2021-03-11 Thread Matthias van de Meent
On Thu, 11 Mar 2021 at 17:31, Robert Haas wrote: > > On Tue, Mar 9, 2021 at 3:35 PM Peter Geoghegan wrote: > > Speaking of line pointer bloat (and "irreversible" bloat), I came > > across something relevant today. I believe that this recent patch from > > Matthias van de Meent is a relatively

Re: pgbench - add pseudo-random permutation function

2021-03-11 Thread David Bowen
The algorithm for generating a random permutation with a uniform distribution across all permutations is easy: for (i=0; i wrote: > On Thu, 11 Mar 2021 at 00:58, Bruce Momjian wrote: > > > > Maybe Dean Rasheed can help because of his math background --- CC'ing > him. > > > > Reading the thread I

Re: A qsort template

2021-03-11 Thread Andres Freund
Hi, I wish we had the same for bsearch... :) On 2021-03-03 17:17:13 +1300, Thomas Munro wrote: > As for which cases are actually worth specialising, I've attached the > example that Andres mentioned earlier; it seems like a reasonable > candidate to go ahead and commit too, but I realised that

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-11 Thread Tom Lane
Alvaro Herrera writes: > [ v35-libpq-pipeline.patch ] I think the changes in pqParseInput3() are broken. You should have kept the else-structure as-is and inserted the check for "not really idle" inside the else-clause that reports an error. As it stands, after successfully processing an

Re: Performance degradation of REFRESH MATERIALIZED VIEW

2021-03-11 Thread Andres Freund
Hi, On 2021-03-11 17:44:37 +0900, Masahiko Sawada wrote: > The execution time of REFRESH MATERIALIZED VIEW are: > > w/ HEAP_INSERT_FROZEN flag : 42 sec > w/o HEAP_INSERT_FROZEN flag : 33 sec > > After investigation, I found that such performance degradation happens > on only HEAD code. It seems

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-11 Thread Tom Lane
The buildfarm has revealed that this patch doesn't work under CLOBBER_CACHE_ALWAYS: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=husky=2021-03-10%2021%3A09%3A32 I initially thought that that was a problem with c3ffe34863, but after reproducing it here I get this stack trace: #0

Re: Huge memory consumption on partitioned table with FKs

2021-03-11 Thread Tom Lane
I wrote: > Now, maybe it's a coincidence that husky failed on a > partitioned-foreign-key test right after this patch went in, but I bet > not. Since husky runs CLOBBER_CACHE_ALWAYS, it looks to me like we've > overlooked some cache-reset scenario or other. After reproducing it here, that *is* a

Re: pg_amcheck contrib application

2021-03-11 Thread Mark Dilger
> On Mar 11, 2021, at 9:10 AM, Andrey Borodin wrote: > > > >> 11 марта 2021 г., в 20:30, Mark Dilger >> написал(а): >> >> >> pg_amcheck does not need a local data directory to check a remote database >> server, though it does need to connect to that server. > No, I mean it it would be

Re: pg_amcheck contrib application

2021-03-11 Thread Andrey Borodin
> 11 марта 2021 г., в 20:30, Mark Dilger > написал(а): > > > pg_amcheck does not need a local data directory to check a remote database > server, though it does need to connect to that server. No, I mean it it would be great if we did not need to materialise whole DB anywhere. Let's say I

Re: Huge memory consumption on partitioned table with FKs

2021-03-11 Thread Tom Lane
Tatsuro Yamada writes: > Thanks for fixing the problem! :-D Hmm, I'm not sure we're done with this patch: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=husky=2021-03-10%2021%3A09%3A32 The critical log extract is 2021-03-11 05:10:13.012 CET [21574:1082] pg_regress/foreign_key LOG:

Re: pgbench - add pseudo-random permutation function

2021-03-11 Thread Dean Rasheed
On Thu, 11 Mar 2021 at 00:58, Bruce Momjian wrote: > > Maybe Dean Rasheed can help because of his math background --- CC'ing him. > Reading the thread I can see how such a function might be useful to scatter non-uniformly random values. The implementation looks plausible too, though it adds

Re: New IndexAM API controlling index vacuum strategies

2021-03-11 Thread Robert Haas
On Tue, Mar 9, 2021 at 3:35 PM Peter Geoghegan wrote: > Speaking of line pointer bloat (and "irreversible" bloat), I came > across something relevant today. I believe that this recent patch from > Matthias van de Meent is a relatively easy way to improve the > situation: > >

Re: Self-join optimisation

2021-03-11 Thread Hywel Carver
Great! It looks like it's been in commitfest status for a few years. Is there anything someone like me (outside the pgsql-hackers community) can do to help it get reviewed this time around? On Thu, Mar 11, 2021 at 2:32 PM Matthias van de Meent < boekewurm+postg...@gmail.com> wrote: > On Thu, 11

Re: Disallow cancellation of waiting for synchronous replication

2021-03-11 Thread Andrey Borodin
Thanks for looking into this! > 11 марта 2021 г., в 19:15, Fujii Masao > написал(а): > > > > On 2020/12/09 18:07, Andrey Borodin wrote: >>> 9 июня 2020 г., в 23:32, Jeff Davis написал(а): >>> >>> >> After using a patch for a while it became obvious that PANICing during >> termination is

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-11 Thread Alvaro Herrera
Rebase to current sources, to appease CF bot; no other changes. -- Álvaro Herrera39°49'30"S 73°17'W diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..ea3ae56991 100644 --- a/doc/src/sgml/catalogs.sgml +++

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
On Wed, Mar 10, 2021 at 11:02 PM Mark Dilger wrote: > > The documentation says that -D "does exclude any database that was > > listed explicitly as dbname on the command line, nor does it exclude > > the database chosen in the absence of any dbname argument." The first > > part of this makes

Re: documentation fix for SET ROLE

2021-03-11 Thread David G. Johnston
On Thu, Mar 11, 2021 at 7:58 AM Laurenz Albe wrote: > I think we should *not* document that under "server configuration". > This is confusing and will lead people to think that a role is > a configuration parameter. But you cannot add > >role = myrole > > to "postgresql.conf". A role is

Re: Autovacuum on partitioned table (autoanalyze)

2021-03-11 Thread David Steele
On 12/14/20 8:46 PM, yuzuko wrote: On Thu, Dec 3, 2020 at 10:28 PM Alvaro Herrera wrote: Attach the new patch based on his patch. What do you think? Álvaro, Justin, Kyotaro, thoughts on this latest patch? Regards, -- -David da...@pgmasters.net

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-11 Thread Alvaro Herrera
On 2021-Mar-05, Alvaro Herrera wrote: > I'll take the weekend to think about the issue with conn->last_query and > conn->queryclass that I mentioned yesterday; other than that detail my > feeling is that this is committable, so I'll be looking at getting this > pushed early next weeks, barring

Re: MultiXact\SLRU buffers configuration

2021-03-11 Thread Gilles Darold
Le 15/02/2021 à 18:17, Andrey Borodin a écrit : 23 дек. 2020 г., в 21:31, Gilles Darold написал(а): Sorry for the response delay, we have run several others tests trying to figure out the performances gain per patch but unfortunately we have very heratic results. With the same parameters

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
On Thu, Mar 11, 2021 at 3:12 AM Peter Eisentraut wrote: > (I'm still not a fan of adding more client-side tools whose sole task is > to execute server-side functionality in a slightly filtered way, but it > seems people are really interested in this, so ...) > > I want to register, if we are

Re: Self-join optimisation

2021-03-11 Thread Justin Pryzby
On Thu, Mar 11, 2021 at 03:32:16PM +0100, Matthias van de Meent wrote: > On Thu, 11 Mar 2021 at 15:15, Hywel Carver wrote: > > I asked this question in the Postgres Slack, and was recommended to ask > > here instead. > > > > A few times, I've been in a situation where I want to join a table to

Re: pg_amcheck contrib application

2021-03-11 Thread Mark Dilger
> On Mar 11, 2021, at 3:36 AM, Andrey Borodin wrote: > > > >> 11 марта 2021 г., в 13:12, Peter Eisentraut >> написал(а): >> >> client-side tools whose sole task is to execute server-side functionality in >> a slightly filtered way > > By the way, can we teach pg_amcheck to verify

Re: non-HOT update not looking at FSM for large tuple update

2021-03-11 Thread John Naylor
On Thu, Mar 11, 2021 at 9:46 AM Matthias van de Meent < boekewurm+postg...@gmail.com> wrote: > Regarding the 2% slack logic, could we change it to use increments of > line pointers instead? That makes it more clear what problem this > solution is trying to work around; that is to say line

Re: pg_amcheck contrib application

2021-03-11 Thread Mark Dilger
> On Mar 11, 2021, at 12:12 AM, Peter Eisentraut > wrote: > > (I'm still not a fan of adding more client-side tools whose sole task is to > execute server-side functionality in a slightly filtered way, but it seems > people are really interested in this, so ...) > > I want to register, if

Re: documentation fix for SET ROLE

2021-03-11 Thread Laurenz Albe
I have had a look at the patch, and while I agree that this should be documented, I am not happy with the patch as it is. I think we should *not* document that under "server configuration". This is confusing and will lead people to think that a role is a configuration parameter. But you cannot

Re: Logical Replication - improve error message while adding tables to the publication in check_publication_add_relation

2021-03-11 Thread Bharath Rupireddy
On Wed, Mar 10, 2021 at 5:48 PM Euler Taveira wrote: > > On Wed, Mar 10, 2021, at 2:14 AM, Bharath Rupireddy wrote: > > While providing thoughts on [1], I observed that the error messages > that are emitted while adding foreign, temporary and unlogged tables > can be improved a bit from the

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-03-11 Thread Fujii Masao
On 2021/03/11 21:29, Masahiro Ikeda wrote: On 2021-03-11 11:52, Fujii Masao wrote: On 2021/03/11 9:38, Masahiro Ikeda wrote: On 2021-03-10 17:08, Fujii Masao wrote: On 2021/03/10 14:11, Masahiro Ikeda wrote: On 2021-03-09 17:51, Fujii Masao wrote: On 2021/03/05 8:38, Masahiro Ikeda

Re: Self-join optimisation

2021-03-11 Thread Matthias van de Meent
On Thu, 11 Mar 2021 at 15:15, Hywel Carver wrote: > > Hi, > > I asked this question in the Postgres Slack, and was recommended to ask here > instead. > > A few times, I've been in a situation where I want to join a table to itself > on its primary key. That typically happens because I have some

Re: Disallow cancellation of waiting for synchronous replication

2021-03-11 Thread Fujii Masao
On 2020/12/09 18:07, Andrey Borodin wrote: 9 июня 2020 г., в 23:32, Jeff Davis написал(а): After using a patch for a while it became obvious that PANICing during termination is not a good idea. Even when we wait for synchronous replication. It generates undesired coredumps. I think

Re: WIP: System Versioned Temporal Table

2021-03-11 Thread Surafel Temesgen
On Wed, Mar 10, 2021 at 9:02 AM Vik Fearing wrote: > > I have plenty of objection. I'm sorry that I am taking so long with my > review. I am still working on it and it is coming soon, I promise. > > okay take your time regards Surafel

Self-join optimisation

2021-03-11 Thread Hywel Carver
Hi, I asked this question in the Postgres Slack, and was recommended to ask here instead. A few times, I've been in a situation where I want to join a table to itself on its primary key. That typically happens because I have some kind of summary view, which I then want to join to the original

Re: non-HOT update not looking at FSM for large tuple update

2021-03-11 Thread Matthias van de Meent
On Tue, 9 Mar 2021 at 18:39, John Naylor wrote: > > I wrote: > > > That seems like the proper fix, and I see you've started a thread for that. > > I don't think that change in behavior would be backpatchable, but patch > > here might have a chance at that. > > I remembered after the fact that

  1   2   >