Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-08 Thread Ashutosh Bapat
On Fri, Jun 8, 2018 at 1:52 AM, Tom Lane wrote > > I'm still of the opinion that find_appinfos_by_relids() needs to be > nuked from orbit. It has nothing to recommend it either from the > standpoint of performance or that of intellectual coherency (or maybe > that problem is just inadequate

Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-08 Thread Robert Haas
On Fri, Jun 8, 2018 at 12:56 AM, Tom Lane wrote: > The pre-v11 incarnation of those functions took a single AppendRelInfo, > specifying an exact translation from one parent relid to one child > relid. The fundamental problem I've got with the current code, entirely > independently of any

Re: Needless additional partition check in INSERT?

2018-06-08 Thread Alvaro Herrera
On 2018-Jun-07, David Rowley wrote: > On 7 June 2018 at 15:57, Alvaro Herrera wrote: > > Hm I was thinking this new function would be companion to ExecConstrains > > (a fact I used in the name I proposed,) so it'd be in the same file > > (probably right after it.) > > Okay. v5 (attached) does

Re: Bug in either collation docs or code

2018-06-08 Thread Melanie Plageman
> > ​Data, apparently...I got the same non-error result before inserting a > record into test1 then I got the expected error. > > Its the function/operator the fails when faced with invalid input, not the > planner, so the error requires data to provoke. > > David J. > > > I tried inserting data

Re: Bug in either collation docs or code

2018-06-08 Thread David G. Johnston
On Fri, Jun 8, 2018 at 9:24 AM, Melanie Plageman wrote: > It seems like this would not allow the function/operator to decide if it > cares about a determinate collation during execution, since it would > already have errored out during planning. > In the case where the function/operator doesn't

Re: Bug in either collation docs or code

2018-06-08 Thread David G. Johnston
On Fri, Jun 8, 2018 at 9:12 AM, Melanie Plageman wrote: > I tried inserting data and did not get an error: > > CREATE TABLE test1 ( > a text COLLATE "de_DE", > b text COLLATE "es_ES" > ); > > INSERT INTO test1 VALUES('b','b'), ('c','c'), ('g','g'), ('h','h'); > SELECT a < (select 'foo'

Re: Bug in either collation docs or code

2018-06-08 Thread Melanie Plageman
> IIRC this was an intentional decision, made on the grounds that we > can't tell whether the function/operator actually cares about having > a determinate collation or not, so we have to leave it to execution of > that function/operator to complain or not. > > In this case, why treat implicit and

Re: Transform for pl/perl

2018-06-08 Thread Tom Lane
I wrote: > ... So if we think that \undef ought to > produce a SQL null, the thing to do is move the dereferencing loop to > the beginning of plperl_sv_to_datum, and then \undef would produce NULL > whether this transform is installed or not. I don't have a well-informed > opinion on whether

Re: Concurrency bug in UPDATE of partition-key

2018-06-08 Thread Dilip Kumar
On Tue, Jun 5, 2018 at 8:03 PM, Amit Khandekar wrote: > Attached is a rebased patch version. Also included it in the upcoming > commitfest : > https://commitfest.postgresql.org/18/1660/ > > In the rebased version, the new test cases are added in the existing >

Re: why partition pruning doesn't work?

2018-06-08 Thread David Rowley
On 8 June 2018 at 18:14, David Rowley wrote: > On 8 June 2018 at 15:22, Tom Lane wrote: >> David Rowley writes: >>> On 8 June 2018 at 03:43, Tom Lane wrote: Maybe there's something I'm missing here, but I sort of hoped that this patch would nuke all the special-case code for Params

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread Tom Lane
Phil Florent writes: > explain analyze select * from v where v.k1 > date '2017-01-01'; > ERREUR: XX000: did not find all requested child rels in append_rel_list > EMPLACEMENT : find_appinfos_by_relids, prepunion.c : 2643 Reproduced here, thanks for the report! This is very timely since we were

Re: Needless additional partition check in INSERT?

2018-06-08 Thread Alvaro Herrera
On 2018-Jun-09, David Rowley wrote: > On 9 June 2018 at 03:24, Alvaro Herrera wrote: > > I was also wondering about introducing a new function call in this path > > where previously was none. Given the amount of other stuff that's > > happening when a tuple is inserted, I suppose it's not worth

Re: Bug in either collation docs or code

2018-06-08 Thread Melanie Plageman
> > I did my test on 9.6.5 ​(Ubuntu 16.04) with: > > CREATE TABLE test_col ( > a text COLLATE "en_CA.utf8", > b text COLLATE "en_US.utf8" > ); > INSERT INTO test_col VALUES ('A', 'A'); > SELECT a < (SELECT 'foo'::text COLLATE "en_GB.utf8") FROM test_col; > > SQL Error: ERROR: could not

Re: Bug in either collation docs or code

2018-06-08 Thread Tom Lane
Melanie Plageman writes: > In this case, why treat implicit and explicit collation conflicts > differently? Um ... because the SQL standard says so? regards, tom lane

Re: Needless additional partition check in INSERT?

2018-06-08 Thread David Rowley
On 9 June 2018 at 03:24, Alvaro Herrera wrote: > I was also wondering about introducing a new function call in this path > where previously was none. Given the amount of other stuff that's > happening when a tuple is inserted, I suppose it's not worth worrying > about in terms of making this an

Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-08 Thread Ashutosh Bapat
On Fri, Jun 8, 2018 at 10:26 AM, Tom Lane wrote: > David Rowley writes: >> On 8 June 2018 at 08:22, Tom Lane wrote: >>> I'm still of the opinion that find_appinfos_by_relids() needs to be >>> nuked from orbit. > >> Yeah, I agree it's not nice that it pallocs an array then pfrees it >> again.

Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread Phil Florent
Hi, I obtained an XX000 error testing my DSS application with PostgreSQL 11 beta 1. Here is a simplified version of my test, no data in the tables : -- 11 select version(); version

Re: High CPU load caused by the autovacuum launcher process

2018-06-08 Thread Jeff Janes
On Fri, Jun 8, 2018 at 3:24 AM, Owayss Kabtoul wrote: > Hi folks, > > I ran into an issue where, on Postgres instances that have a very large > number of databases per cluster (~15K), the autovacuum process seems to > have a very high impact on CPU usage. Specifically, it is the autovacuum >

Re: why partition pruning doesn't work?

2018-06-08 Thread David Rowley
On 8 June 2018 at 15:22, Tom Lane wrote: > David Rowley writes: >> On 8 June 2018 at 03:43, Tom Lane wrote: >>> Maybe there's something I'm missing here, but I sort of hoped that this >>> patch would nuke all the special-case code for Params in this area. >>> Why is there any need to

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Amit Kapila
On Fri, Jun 8, 2018 at 2:55 AM, Andres Freund wrote: > > On 2018-06-07 14:19:18 -0700, Andres Freund wrote: > > Hi, > > > > On 2018-03-29 12:17:24 +0100, Greg Stark wrote: > > > I'm poking around to see debug a vacuuming problem and wondering if > > > I've found something more serious. > > > > >

High CPU load caused by the autovacuum launcher process

2018-06-08 Thread Owayss Kabtoul
Hi folks, I ran into an issue where, on Postgres instances that have a very large number of databases per cluster (~15K), the autovacuum process seems to have a very high impact on CPU usage. Specifically, it is the autovacuum launcher process, not the workers. The launcher process eats a whole

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Simon Riggs
On 7 June 2018 at 22:25, Andres Freund wrote: > On 2018-06-07 14:19:18 -0700, Andres Freund wrote: > Look at: > > void > ProcArrayApplyRecoveryInfo(RunningTransactions running) > ... > /* > * Remove stale locks, if any. > * > * Locks are always assigned to the

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Amit Kapila
On Fri, Jun 8, 2018 at 1:53 PM, Simon Riggs wrote: > > So the attached patch fixes both the bug in the recent commit and the > one I just found by observation of 49bff5300d527, since they are > related. > > StandbyReleaseOldLocks() can sweep in the same way as >

Re: Concurrency bug in UPDATE of partition-key

2018-06-08 Thread Dilip Kumar
On Fri, Jun 8, 2018 at 2:23 PM, Dilip Kumar wrote: > On Tue, Jun 5, 2018 at 8:03 PM, Amit Khandekar > wrote: > >> Attached is a rebased patch version. Also included it in the upcoming >> commitfest : >> https://commitfest.postgresql.org/18/1660/ >> >> In the rebased version, the new test cases

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Andres Freund
On 2018-06-08 11:29:17 +0530, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 2:55 AM, Andres Freund wrote: > > > > On 2018-06-07 14:19:18 -0700, Andres Freund wrote: > > > We currently do acquire an xid when truncating the relation - but I > > > think it'd somewhat fair to argue that that's somewhat

Re: file cloning in pg_upgrade and CREATE DATABASE

2018-06-08 Thread Robert Haas
On Wed, Jun 6, 2018 at 11:58 AM, Peter Eisentraut wrote: > --reflink={always,auto,never}. (This option name is adapted from GNU ... > The setting always requires the use of relinks. If Is it supposed to be relinks or reflinks? The two lines above don't agree. -- Robert Haas EnterpriseDB:

Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-08 Thread Tom Lane
Robert Haas writes: > That being said, I don't mind a bit if you want to look for further > speedups here, but if you do, keep in mind that a lot of queries won't > even use partition-wise join, so all of the arrays will be of length > 1. Even when partition-wise join is used, it is quite likely

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Peter Da Silva
On 6/8/18, 2:21 PM, "Andres Freund" wrote: Not sure I quite understand what you mean. You're thinking of the case where you're processing rows one-by-one with a cursor? Or that a single spi call takes a long while to process the query? The former, I believe. One example (lightly

Re: Loaded footgun open_datasync on Windows

2018-06-08 Thread Magnus Hagander
On Fri, Jun 8, 2018 at 4:55 AM, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 7:48 AM, Laurenz Albe > wrote: > >> Amit Kapila wrote: >> > On Wed, Jun 6, 2018 at 3:06 PM, Kuntal Ghosh < >> kuntalghosh.2...@gmail.com> wrote: >> > > It seems the "#ifndef FRONTEND" restriction was added around >> > >

pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Peter Da Silva
We have occasional need to run very long-running pl/tcl scripts. If the request is cancelled (say, by the user hitting ^c in psql) the server-side script still runs to completion. There is a C-level variable QueryCancelPending that can be used to monitor for this case, but it’s not visible at

Re: Needless additional partition check in INSERT?

2018-06-08 Thread Alvaro Herrera
On 2018-Jun-09, David Rowley wrote: > Of course, that could be fixed by adding something like "bool > isinsert" to ExecConstraints(), so that it does not do the needless > check on UPDATE, Yeah, that was my actual suggestion rather than using Amit's patch verbatim. > but I'm strongly against

Re: Needless additional partition check in INSERT?

2018-06-08 Thread Alvaro Herrera
On 2018-Jun-09, David Rowley wrote: > Of course, that could be fixed by adding something like "bool > isinsert" to ExecConstraints(), so that it does not do the needless > check on UPDATE, Yeah, that was my actual suggestion rather than using Amit's patch verbatim. > but I'm strongly against

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Tom Lane
Andres Freund writes: > On 2018-06-08 18:08:14 +, Peter Da Silva wrote: >> There is a C-level variable QueryCancelPending that can be used to >> monitor for this case, but it’s not visible at the pl/tcl scripting >> level. This is a simple new command that returns the current state of >> this

Re: SHOW ALL does not honor pg_read_all_settings membership

2018-06-08 Thread Alvaro Herrera
On 2018-Jun-08, Alvaro Herrera wrote: > BTW a further bug here seems to be that "select * from pg_settings()" > does not show the source file/line for members of the role, which seems > to be documented to occur. And I think this fixes it. -- Álvaro Herrera

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Tom Lane
Andres Freund writes: > Either way, I'm not convinced that handling query cancels in isolation > is really the right thing. I think pretty much all forms of interrupt > would need to be processed, not just cancels. +1 regards, tom lane

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Andres Freund
Hi, On 2018-06-08 18:08:14 +, Peter Da Silva wrote: > We have occasional need to run very long-running pl/tcl scripts. If > the request is cancelled (say, by the user hitting ^c in psql) the > server-side script still runs to completion. > > There is a C-level variable QueryCancelPending

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Peter Da Silva
On 6/8/18, 1:12 PM, "Andres Freund" wrote: I'm not terribly opposed to this, but I wonder if the much more pragmatic solution is to just occasionally call a database function that checks this? You could just run SELECT 1 occasionally :/ That seems to work, and I suppose in most

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Peter Da Silva
On 6/8/18, 1:12 PM, "Andres Freund" wrote: I'm not terribly opposed to this, but I wonder if the much more pragmatic solution is to just occasionally call a database function that checks this? You could just run SELECT 1 occasionally :/ After further discussion with our team: Would

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Andres Freund
Hi, On 2018-06-08 09:23:02 +0100, Simon Riggs wrote: > I have also found another bug which affects what we do next. > > For context, AEL locks are normally removed by COMMIT or ABORT. > StandbyReleaseOldLocks() is just a sweeper to catch anything that > didn't send an abort before it died, so it

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread David Rowley
On 9 June 2018 at 04:57, Tom Lane wrote: > Phil Florent writes: >> explain analyze select * from v where v.k1 > date '2017-01-01'; >> ERREUR: XX000: did not find all requested child rels in append_rel_list >> EMPLACEMENT : find_appinfos_by_relids, prepunion.c : 2643 > > Reproduced here, thanks

Re: SHOW ALL does not honor pg_read_all_settings membership

2018-06-08 Thread Alvaro Herrera
BTW a further bug here seems to be that "select * from pg_settings()" does not show the source file/line for members of the role, which seems to be documented to occur. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training &

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Andres Freund
On 2018-06-08 14:41:41 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-06-08 18:08:14 +, Peter Da Silva wrote: > >> There is a C-level variable QueryCancelPending that can be used to > >> monitor for this case, but it’s not visible at the pl/tcl scripting > >> level. This is a

Re: pl/tcl function to detect when a request has been canceled

2018-06-08 Thread Andres Freund
On 2018-06-08 19:16:49 +, Peter Da Silva wrote: > On 6/8/18, 1:12 PM, "Andres Freund" wrote: > I'm not terribly opposed to this, but I wonder if the much more > pragmatic solution is to just occasionally call a database function that > checks this? You could just run SELECT 1

Re: Bug in either collation docs or code

2018-06-08 Thread Tom Lane
Melanie Plageman writes: > On postgres built off of master on my mac (sierra), the following is the > output: [ scratches head ... ] I get the same results on either Mac or Linux: regression=# create database u8 encoding utf8 template template0; CREATE DATABASE regression=# \c u8 You are now

Re: Needless additional partition check in INSERT?

2018-06-08 Thread David Rowley
On 9 June 2018 at 04:52, Alvaro Herrera wrote: > Truth is, the more I look at this, the more I think it should be done in > the way Amit Langote was proposing: do away with the extra function, and > check all those conditions inside ExecConstraints itself. We can add a > new boolean tupleRouting

Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-08 Thread Tom Lane
Robert Haas writes: > On Fri, Jun 8, 2018 at 12:56 AM, Tom Lane wrote: >> The pre-v11 incarnation of those functions took a single AppendRelInfo, >> specifying an exact translation from one parent relid to one child >> relid. The fundamental problem I've got with the current code, entirely >>

Re: Compromised postgresql instances

2018-06-08 Thread Andrew Dunstan
On 06/08/2018 06:13 PM, Andrew Gierth wrote: "Tom" == Tom Lane writes: > Andrew Dunstan writes: >> Please cite actual instances of such reports. Vague queries like >> this help nobody. We do also get them on the IRC channel every once in a while, not very frequently but enough to

Re: Transform for pl/perl

2018-06-08 Thread Tom Lane
I wrote: > I'm inclined to think that auto-dereference is indeed a good idea, > and am tempted to go make that change to make all this consistent. > Comments? Here's a draft patch for that. I ended up only changing plperl_sv_to_datum. There is maybe a case for doing something similar in

Re: Compromised postgresql instances

2018-06-08 Thread Andrew Dunstan
On 06/08/2018 04:54 PM, Steve Atkins wrote: On Jun 8, 2018, at 1:47 PM, Tom Lane wrote: Andrew Dunstan writes: On 06/08/2018 04:34 PM, Steve Atkins wrote: I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with

Re: SHOW ALL does not honor pg_read_all_settings membership

2018-06-08 Thread Alvaro Herrera
On 2018-Mar-01, Laurenz Albe wrote: > I noticed that commit 25fff40798fc4ac11a241bfd9ab0c45c085e2212 forgot > to teach SHOW ALL to show all GUCs when the user belongs to > pg_read_all_settings. > > Patch attached; I think this should be backpatched. Done, with the further fixes downthread.

Re: Compromised postgresql instances

2018-06-08 Thread Tom Lane
Andrew Dunstan writes: > On 06/08/2018 04:34 PM, Steve Atkins wrote: >> I've noticed a steady trickle of reports of postgresql servers being >> compromised via being left available to the internet with insecure or >> default configuration, or brute-forced credentials. The symptoms are >>

Re: Compromised postgresql instances

2018-06-08 Thread Steve Atkins
> On Jun 8, 2018, at 1:47 PM, Tom Lane wrote: > > Andrew Dunstan writes: >> On 06/08/2018 04:34 PM, Steve Atkins wrote: >>> I've noticed a steady trickle of reports of postgresql servers being >>> compromised via being left available to the internet with insecure or >>> default

Re: Compromised postgresql instances

2018-06-08 Thread Andrew Gierth
> "Tom" == Tom Lane writes: > Andrew Dunstan writes: >> Please cite actual instances of such reports. Vague queries like >> this help nobody. We do also get them on the IRC channel every once in a while, not very frequently but enough to notice (maybe 2-3 so far this year?). Tom>

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-08 Thread Alvaro Herrera
On 2018-Jun-08, Alvaro Herrera wrote: > Actually, the column order doesn't matter for a trigger function, > because these don't refer to columns by number but by name. So unless > users write trigger functions in C and use hardcoded column numbers > (extremely unlikely), I think this is not an

Compromised postgresql instances

2018-06-08 Thread Steve Atkins
I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries being uploaded to the data directory and executed with the

Re: Compromised postgresql instances

2018-06-08 Thread Andrew Dunstan
On 06/08/2018 04:34 PM, Steve Atkins wrote: I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries being uploaded

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-08 Thread Alvaro Herrera
On 2018-Jun-07, Ashutosh Bapat wrote: > On Thu, Jun 7, 2018 at 10:58 AM, Amit Langote > wrote: > > I don't understand why you think it's too troublesome to let the users > > know that there is some way to use BR triggers with partitioning. We > > didn't do that for indexes, for example, before

Re: Compromised postgresql instances

2018-06-08 Thread Thomas Kellerer
> Please cite actual instances of such reports. Vague queries like this help nobody. There were several questions on SO https://stackoverflow.com/questions/49815460 https://stackoverflow.com/questions/47499766 https://stackoverflow.com/questions/47741077

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread David Rowley
On 9 June 2018 at 06:50, David Rowley wrote: > It looks like this was 499be013de6, which was one of mine. > > A more simple case to reproduce is: > > drop table listp; > create table listp (a int, b int) partition by list(a); > create table listp1 partition of listp for values in (1); > select *