Re: Optimizer docs typos

2020-05-21 Thread Etsuro Fujita
On Wed, May 20, 2020 at 7:17 PM Etsuro Fujita wrote: > On Tue, May 19, 2020 at 7:35 PM Etsuro Fujita wrote: > > On Mon, May 18, 2020 at 7:45 PM Richard Guo wrote: > > > In this same README doc, another suspicious typo to me, which happens in > > > section "Optimizer Functions", is in the prefix

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-21 Thread Amit Kapila
On Tue, May 19, 2020 at 6:01 PM Amit Kapila wrote: > > On Fri, May 15, 2020 at 2:48 PM Dilip Kumar wrote: > > I have further reviewed v22 and below are my comments: v22-0005-Implement-streaming-mode-in-ReorderBuffer -- 1. +

Re: Problem with pg_atomic_compare_exchange_u64 at 32-bit platforms

2020-05-21 Thread Noah Misch
On Wed, May 20, 2020 at 10:59:44AM +0300, Konstantin Knizhnik wrote: > On 20.05.2020 10:36, Noah Misch wrote: > >On Wed, May 20, 2020 at 10:23:37AM +0300, Konstantin Knizhnik wrote: > >>On 20.05.2020 06:05, Noah Misch wrote: > >>>On Tue, May 19, 2020 at 04:07:29PM +0300, Konstantin Knizhnik wrote:

Re: [Proposal] Page Compression for OLTP

2020-05-21 Thread chenhj
Sorry, There may be a problem with the display format of the previous mail. So resend it At 2020-05-21 15:04:55, "Fabien COELHO" wrote: > >Hello, > >My 0.02, some of which may just show some mi

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-21 Thread Andy Fan
On Thu, May 21, 2020 at 3:49 PM Julien Rouhaud wrote: > Le jeu. 21 mai 2020 à 09:17, Michael Paquier a > écrit : > >> On Thu, May 21, 2020 at 08:49:53AM +0200, Julien Rouhaud wrote: >> > On Tue, May 19, 2020 at 4:29 AM Andy Fan >> wrote: >> >> Thanks for the excellent extension. I want to add 5

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-21 Thread Andy Fan
On Thu, May 21, 2020 at 3:17 PM Michael Paquier wrote: > On Thu, May 21, 2020 at 08:49:53AM +0200, Julien Rouhaud wrote: > > On Tue, May 19, 2020 at 4:29 AM Andy Fan > wrote: > >> Thanks for the excellent extension. I want to add 5 more fields to > satisfy the > >> following requirements. > >> >

Re: POC: rational number type (fractions)

2020-05-21 Thread Noah Misch
On Thu, May 21, 2020 at 01:40:10PM -0400, Robert Haas wrote: > On Mon, May 18, 2020 at 6:15 PM Tom Lane wrote: > > There surely are use-cases for true rational arithmetic, but I'm > > dubious that it belongs in core Postgres. I don't think that enough > > of our users would want it to justify exp

Re: race condition when writing pg_control

2020-05-21 Thread Thomas Munro
On Tue, May 5, 2020 at 9:51 AM Thomas Munro wrote: > On Tue, May 5, 2020 at 5:53 AM Bossart, Nathan wrote: > > I believe I've discovered a race condition between the startup and > > checkpointer processes that can cause a CRC mismatch in the pg_control > > file. If a cluster crashes at the right

More tests with USING INDEX replident and dropped indexes

2020-05-21 Thread Michael Paquier
Hi all, While working on some other logical decoding patch recently, I bumped into the fact that we have special handling for the case of REPLICA IDENTITY USING INDEX when the dependent index is dropped, where the code handles that case as an equivalent of NOTHING. Attached is a patch to add more

Re: Parallel Seq Scan vs kernel read ahead

2020-05-21 Thread Thomas Munro
On Fri, May 22, 2020 at 1:14 PM Soumyadeep Chakraborty wrote: > Some more data points: Thanks! > max_parallel_workers_per_gatherTime(seconds) > 0 29.04s > 1 29.17s > 2 28.7

Re: Parallel Seq Scan vs kernel read ahead

2020-05-21 Thread Soumyadeep Chakraborty
Hi Thomas, Some more data points: create table t_heap as select generate_series(1, 1) i; Query: select count(*) from t_heap; shared_buffers=32MB (so that I don't have to clear buffers, OS page cache) OS: FreeBSD 12.1 with UFS on GCP 4 vCPUs, 4GB RAM Intel Skylake 22G Google PersistentDis

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-05-21 Thread Andy Fan
> My question is whether it should be added as an optional facility of a > parameterised sub plan, rather than an always-needed full-strength node. > That way the choice of whether to use it can happen at execution time once > we notice that we've been called too many times. > > Actually I am not

Re: [Proposal] Page Compression for OLTP

2020-05-21 Thread chenhj
At 2020-05-21 15:04:55, "Fabien COELHO" wrote: > >Hello, > >My 0.02, some of which may just show some misunderstanding on my part: > > - Could this be proposed as some kind of extension, provided that enough > hooks are available? ISTM that foreign tables and/or alternative > storage engine (

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-05-21 Thread David Rowley
On Thu, 21 May 2020 at 00:56, Simon Riggs wrote: > I thought the main reason to do this was the case when the nested loop > subplan was significantly underestimated and we realize during execution that > we should have built a hash table. So including this based on cost alone > seems to miss a

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread Andy Fan
On Fri, May 22, 2020 at 4:52 AM David Rowley wrote: > On Thu, 14 May 2020 at 14:39, Andy Fan wrote: > > > > On Thu, May 14, 2020 at 6:20 AM David Rowley > wrote: > >> Having the "onerow" flag was not how I intended it to work. > >> > > Thanks for the detailed explanation. So I think we do need

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread Andy Fan
> if I understand correctly those two are introducing the concept, and others are just using it You are understand it correctly. -- Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread Andy Fan
On Fri, May 22, 2020 at 4:40 AM David Rowley wrote: > On Fri, 22 May 2020 at 07:49, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > * It seems populate_baserel_uniquekeys, which actually sets uniquekeys, > > is called after create_index_paths, where index skip scan already > > needs to use t

Re: pgindent && weirdness

2020-05-21 Thread Tom Lane
Piotr Stefaniak writes: > On 16/01/2020 03.59, Thomas Munro wrote: >> One way to fix that in the cases Alvaro is referring to is to tell >> override the setting so that && (and likewise ||) are never considered >> to be unary, though I haven't tested this much and there are surely >> other ways

Re: Parallel Seq Scan vs kernel read ahead

2020-05-21 Thread Thomas Munro
On Fri, May 22, 2020 at 10:00 AM David Rowley wrote: > On Thu, 21 May 2020 at 17:06, David Rowley wrote: > > For the patch. I know you just put it together quickly, but I don't > > think you can do that ramp up the way you have. It looks like there's > > a risk of torn reads and torn writes and I

Re: Parallel Seq Scan vs kernel read ahead

2020-05-21 Thread David Rowley
On Thu, 21 May 2020 at 17:06, David Rowley wrote: > For the patch. I know you just put it together quickly, but I don't > think you can do that ramp up the way you have. It looks like there's > a risk of torn reads and torn writes and I'm unsure how much that > could affect the test results here.

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Thu, May 21, 2020 at 02:16:37PM -0700, Jeff Davis wrote: On Thu, 2020-05-21 at 21:13 +0200, Tomas Vondra wrote: 2) We could make it self-tuning, by increasing the number of blocks we pre-allocate. So every time we exhaust the range, we double the number of blocks (with a reasonable maximum, l

Re: pgindent && weirdness

2020-05-21 Thread Piotr Stefaniak
On 16/01/2020 03.59, Thomas Munro wrote: On Wed, Jan 15, 2020 at 11:30 AM Tom Lane wrote: Alvaro Herrera writes: I just ran pgindent over some patch, and noticed that this hunk ended up in my working tree: - if (IsA(leftop, Var) && IsA(rightop, Const)) + if (IsA(leftop, Var) &&IsA(

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Jeff Davis
On Thu, 2020-05-21 at 21:13 +0200, Tomas Vondra wrote: > 2) We could make it self-tuning, by increasing the number of blocks > we pre-allocate. So every time we exhaust the range, we double the > number of blocks (with a reasonable maximum, like 1024 or so). Or we > might just increment it by 32, o

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Thu, May 21, 2020 at 12:40:23PM -0700, Jeff Davis wrote: On Thu, 2020-05-21 at 21:13 +0200, Tomas Vondra wrote: 1) Instead of assigning the pages one by one, we can easily extend the API to allow getting a range of blocks, so that we don't need to call ltsGetFreeBlock in a loop. Instead we co

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread David Rowley
On Thu, 14 May 2020 at 14:39, Andy Fan wrote: > > On Thu, May 14, 2020 at 6:20 AM David Rowley wrote: >> Having the "onerow" flag was not how I intended it to work. >> > Thanks for the detailed explanation. So I think we do need to handle onerow > specially, (It means more things than adding eac

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread David Rowley
On Fri, 22 May 2020 at 07:49, Dmitry Dolgov <9erthali...@gmail.com> wrote: > * It seems populate_baserel_uniquekeys, which actually sets uniquekeys, > is called after create_index_paths, where index skip scan already > needs to use them. Is it possible to call it earlier? Seems reasonable. I o

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread Dmitry Dolgov
> On Tue, Apr 14, 2020 at 09:09:31PM +1200, David Rowley wrote: > > The infrastructure (knowing the unique properties of a RelOptInfo), as > provided by the patch Andy has been working on, which is based on my > rough prototype version, I believe should be used for the skip scans > patch as well.

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Jeff Davis
On Thu, 2020-05-21 at 21:13 +0200, Tomas Vondra wrote: > 1) Instead of assigning the pages one by one, we can easily extend > the > API to allow getting a range of blocks, so that we don't need to call > ltsGetFreeBlock in a loop. Instead we could call ltsGetFreeBlockRange > with the requested numb

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Thu, May 21, 2020 at 12:04:19PM -0700, Jeff Davis wrote: On Thu, 2020-05-21 at 20:54 +0200, Tomas Vondra wrote: The last column is master with the tlist tweak alone - it's better than hashagg on master alone, but it's not nearly as good as with both tlist and prealloc patches. Right, I cert

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Tue, May 19, 2020 at 09:15:40PM -0700, Jeff Davis wrote: On Tue, 2020-05-19 at 19:53 +0200, Tomas Vondra wrote: And if there a way to pre-allocate larger chunks? Presumably we could assign the blocks to tape in larger chunks (e.g. 128kB, i.e. 16 x 8kB) instead of just single block. I haven't

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Jeff Davis
On Thu, 2020-05-21 at 20:54 +0200, Tomas Vondra wrote: > The last column is master with the tlist tweak alone - it's better > than > hashagg on master alone, but it's not nearly as good as with both > tlist > and prealloc patches. Right, I certainly think we should do the prealloc change, as well.

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Thu, May 21, 2020 at 08:34:05PM +0200, Tomas Vondra wrote: On Thu, May 21, 2020 at 11:19:01AM -0700, Jeff Davis wrote: ... I think we should do the pre-allocation patch too. I haven't tried yet but I believe the tlist fix alone won't do nearly as good. I've done some measurements on the s

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Thu, May 21, 2020 at 11:19:01AM -0700, Jeff Davis wrote: On Thu, 2020-05-21 at 16:30 +0200, Tomas Vondra wrote: OK, it seems the attached trivial fix (simply changing CP_LABEL_TLIST to CP_SMALL_TLIST) addresses this for me. Great! There were a couple plan changes where it introduced a Subq

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Jeff Davis
On Thu, 2020-05-21 at 16:30 +0200, Tomas Vondra wrote: > OK, it seems the attached trivial fix (simply changing CP_LABEL_TLIST > to > CP_SMALL_TLIST) addresses this for me. Great! There were a couple plan changes where it introduced a Subquery Scan. I'm not sure that I understand why it's doing t

Re: POC: rational number type (fractions)

2020-05-21 Thread Robert Haas
On Mon, May 18, 2020 at 6:15 PM Tom Lane wrote: > There surely are use-cases for true rational arithmetic, but I'm > dubious that it belongs in core Postgres. I don't think that enough > of our users would want it to justify expending core-project maintenance > effort on it. So I'd be happier to

Re: pg_bsd_indent and -Wimplicit-fallthrough

2020-05-21 Thread Piotr Stefaniak
On 18/05/2020 11.22, Julien Rouhaud wrote: On Sun, May 17, 2020 at 2:32 AM Michael Paquier wrote: On Sat, May 16, 2020 at 11:56:28AM -0400, Tom Lane wrote: In the meantime, I went ahead and pushed this to our pg_bsd_indent repo. Thanks, Tom. +1, thanks a lot! Committed upstream, thank

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Robert Haas
On Thu, May 21, 2020 at 10:45 AM Tomas Vondra wrote: > So the pre-allocation makes it 10x faster, and the tlist tweak makes it > 5x faster. Not bad, I guess. That is pretty great stuff, Tomas. FWIW, I agree that CP_SMALL_TLIST seems like the right thing here. -- Robert Haas EnterpriseDB: http:

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-21 Thread Pantelis Theodosiou
On Thu, May 21, 2020 at 3:20 PM Pantelis Theodosiou wrote: > Congrats to all for the release of a new major version! > > Two questions: > - Why is VACUUM together with FETCH FIRST WITH TIES, CREATE TABLE LIKE, > ALTER VIEW, ALTER TABLE, etc in Utility Commands section? > Shouldn't there be a se

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Thu, May 21, 2020 at 02:12:55AM +0200, Tomas Vondra wrote: ... I agree that's pretty nice. I wonder how far would we need to go before reaching a plateau. I'll try this on the other machine with temporary tablespace on SATA, but that'll take longer. OK, I've managed to get some numbers fr

Re: Behaviour of failed Primary

2020-05-21 Thread Amit Kapila
On Thu, May 21, 2020 at 5:38 PM Santhosh Kumar wrote: > > Hi Forum, > If I have a cluster with Synchronous replication enabled with three nodes, > for eg: > > [primary] [hot stand by 1] [host stand by 2] > > And for some unforeseen reasons, if primary fails, the failover will kick in > and hot

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Thu, May 21, 2020 at 03:41:22PM +0200, Tomas Vondra wrote: On Tue, May 19, 2020 at 05:12:02PM +0200, Tomas Vondra wrote: ... The problem is that the hashagg plan runs in ~1400 seconds, while the groupagg only takes ~360. And per explain analyze, the difference really is in the aggregation -

Re: [PATCH] fix GIN index search sometimes losing results

2020-05-21 Thread Pavel Borisov
Hi All! 1. Generally the difference of my patch in comparison to Tom's patch 0001 is that I tried to move previous logic of GIN's own TS_execute_ternary() to the general logic of TS_execute_recurse and in case we have index without positions to avoid diving into phrase operator replacing (only in t

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-21 Thread Pantelis Theodosiou
Congrats to all for the release of a new major version! Two questions: - Why is VACUUM together with FETCH FIRST WITH TIES, CREATE TABLE LIKE, ALTER VIEW, ALTER TABLE, etc in Utility Commands section? Shouldn't there be a separate section for SQL changes? (or keep one section but rename the Util

Re: Schedule of commit fests for PG14

2020-05-21 Thread David Steele
On 5/21/20 2:35 AM, Michael Paquier wrote: Hi all, Normally $subject would have been discussed at the developer meeting in Ottawa, but that's not going to happen per the current situation. For the last couple of years, we have been using the same timeline for for commit fests in a development c

Re: Schedule of commit fests for PG14

2020-05-21 Thread Tom Lane
Michael Paquier writes: > Normally $subject would have been discussed at the developer meeting > in Ottawa, but that's not going to happen per the current situation. > For the last couple of years, we have been using the same timeline for > for commit fests in a development cycle, so why not goin

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-21 Thread Tomas Vondra
On Tue, May 19, 2020 at 05:12:02PM +0200, Tomas Vondra wrote: ... The problem is that the hashagg plan runs in ~1400 seconds, while the groupagg only takes ~360. And per explain analyze, the difference really is in the aggregation - if we subtract the seqscan, the sort+groupagg takes about 310s

Re: WIP/PoC for parallel backup

2020-05-21 Thread Robert Haas
On Thu, May 21, 2020 at 2:06 AM Rushabh Lathia wrote: > Yes. My colleague Suraj tried this and here are the pg_stat_activity output > files. > > Captured wait events after every 3 seconds during the backup for - > 1: parallel backup for 100GB data with 4 workers > (pg_stat_activity_normal_backup

Re: factorial function/phase out postfix operators?

2020-05-21 Thread Robert Haas
On Wed, May 20, 2020 at 2:24 PM Tom Lane wrote: > Right; I'd done the same arithmetic. Since we currently have a total > of 450 keywords of all flavors, that means we can make either 64% > of them or 74.6% of them be safe to use as bare column labels. While > that's surely better than today, it

Re: proposal: schema variables

2020-05-21 Thread Pavel Stehule
. Pavel > -- > With Regards, > Amit Kapila. > EnterpriseDB: http://www.enterprisedb.com > schema-variables-20200521.patch.gz Description: application/gzip

Behaviour of failed Primary

2020-05-21 Thread Santhosh Kumar
Hi Forum, If I have a cluster with Synchronous replication enabled with three nodes, for eg: [primary] [hot stand by 1] [host stand by 2] And for some unforeseen reasons, if primary fails, the failover will kick in and hot stand by 1 will become new primary and cluster setup will look like this

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-21 Thread Jonathan S. Katz
Hi John, On 5/21/20 12:12 AM, John Naylor wrote: > Hi Jon, > > I noticed a couple minor inconsistencies: > > ".datetime" -> elsewhere functions are formatted as `.datetime()` > > libpq -> `libpq` > > The link to the release notes on its own line is the same as the > inline link, if that makes

Re: proposal: schema variables

2020-05-21 Thread Amit Kapila
On Thu, May 21, 2020 at 3:41 PM Pavel Stehule wrote: > > Hi > > just rebase without any other changes > You seem to forget attaching the rebased patch. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: proposal: schema variables

2020-05-21 Thread Pavel Stehule
Hi just rebase without any other changes Regards Pavel

Re: pg13: xlogreader API adjust

2020-05-21 Thread Kyotaro Horiguchi
At Fri, 15 May 2020 19:24:28 -0400, Alvaro Herrera wrote in > On 2020-May-15, Michael Paquier wrote: > > > On Thu, May 14, 2020 at 02:12:25PM +0900, Kyotaro Horiguchi wrote: > > > Good catch! That's not only for CreateDecodingContet. That happens > > > everywhere in the query loop in PostgresM

Re: some grammar refactoring

2020-05-21 Thread Rushabh Lathia
On Tue, May 19, 2020 at 12:13 PM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > Here is a series of patches to do some refactoring in the grammar around > the commands COMMENT, DROP, SECURITY LABEL, and ALTER EXTENSION ... > ADD/DROP. In the grammar, these commands (with some excep

Re: Is it useful to record whether plans are generic or custom?

2020-05-21 Thread Tatsuro Yamada
Hi Torikoshi-san! On 2020/05/21 17:10, Kyotaro Horiguchi wrote: At Thu, 21 May 2020 12:18:16 +0900, Fujii Masao wrote in On 2020/05/20 21:56, Atsushi Torikoshi wrote: On Wed, May 20, 2020 at 1:32 PM Kyotaro Horiguchi mailto:horikyota@gmail.com>> wrote: At Tue, 19 May 2020 22:56:17

Re: [bug] Table not have typarray when created by single user mode

2020-05-21 Thread Shawn Wang
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested I verified and found no problems.

Re: Is it useful to record whether plans are generic or custom?

2020-05-21 Thread Kyotaro Horiguchi
At Thu, 21 May 2020 12:18:16 +0900, Fujii Masao wrote in > > > On 2020/05/20 21:56, Atsushi Torikoshi wrote: > > On Wed, May 20, 2020 at 1:32 PM Kyotaro Horiguchi > > mailto:horikyota@gmail.com>> wrote: > > At Tue, 19 May 2020 22:56:17 +0900, Atsushi Torikoshi > > mailto:ato...@gma

Re: [Proposal] Page Compression for OLTP

2020-05-21 Thread Fabien COELHO
Hello, My 0.02€, some of which may just show some misunderstanding on my part: - you have clearly given quite a few thoughts about the what and how… which makes your message an interesting read. - Could this be proposed as some kind of extension, provided that enough hooks are availabl

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-21 Thread Julien Rouhaud
Le jeu. 21 mai 2020 à 09:17, Michael Paquier a écrit : > On Thu, May 21, 2020 at 08:49:53AM +0200, Julien Rouhaud wrote: > > On Tue, May 19, 2020 at 4:29 AM Andy Fan > wrote: > >> Thanks for the excellent extension. I want to add 5 more fields to > satisfy the > >> following requirements. > >> >

Re: explicit_bzero for sslpassword

2020-05-21 Thread Michael Paquier
On Wed, May 20, 2020 at 10:06:55AM +0200, Peter Eisentraut wrote: > Looks correct to me. Thanks for confirming, Peter. Got this one applied. -- Michael signature.asc Description: PGP signature

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-21 Thread Michael Paquier
On Thu, May 21, 2020 at 08:49:53AM +0200, Julien Rouhaud wrote: > On Tue, May 19, 2020 at 4:29 AM Andy Fan wrote: >> Thanks for the excellent extension. I want to add 5 more fields to satisfy >> the >> following requirements. >> >> int subplan; /* No. of subplan in this query */ >> int subque

speed up unicode normalization quick check

2020-05-21 Thread John Naylor
Hi, Attached is a patch to use perfect hashing to speed up Unicode normalization quick check. 0001 changes the set of multipliers attempted when generating the hash function. The set in HEAD works for the current set of NFC codepoints, but not for the other types. Also, the updated multipliers no