performance regression when filling in a table

2019-04-29 Thread Fabien COELHO
Hello devs, On my SSD Ubuntu laptop, with postgres-distributed binaries and unmodified default settings using local connections: ## pg 11.2 > time pgbench -i -s 100 ... done in 31.51 s # (drop tables 0.00 s, create tables 0.01 s, generate 21.30 s, vacuum 3.32 s, primary keys 6.88

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-04-29 Thread Tom Lane
Andres Freund writes: > On April 29, 2019 9:37:33 PM PDT, Tom Lane wrote: >> Seems like putting reindexes of pg_class into a test script that runs >> in parallel with other DDL wasn't a hot idea. > Saw that. Will try to reproduce (and if necessary either run separately or > revert). But isn't

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-04-29 Thread Andres Freund
Hi, On April 29, 2019 9:37:33 PM PDT, Tom Lane wrote: >Andres Freund writes: >> I've pushed the master bits, and the other branches are running >> check-world right now and I'll push soon unless something breaks >(it's a >> bit annoying that <= 9.6 can't run check-world in parallel...). >

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-04-29 Thread Tom Lane
Andres Freund writes: > I've pushed the master bits, and the other branches are running > check-world right now and I'll push soon unless something breaks (it's a > bit annoying that <= 9.6 can't run check-world in parallel...). Seems like putting reindexes of pg_class into a test script that

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-04-29 Thread Andres Freund
Hi, On 2019-04-29 15:09:24 -0700, Andres Freund wrote: > On 2019-04-29 18:07:07 -0400, Tom Lane wrote: > > I wrote: > > > Andres Freund writes: > > >> Taking this as a WIP, what do you think? > > > > > Seems generally about right. > > > > Andres, are you pushing this forward? Next week's

Re: Caveats from reloption toast_tuple_target

2019-04-29 Thread David Rowley
On Tue, 16 Apr 2019 at 23:30, David Rowley wrote: > I've attached a patch which increases the lower limit up to > TOAST_TUPLE_TARGET. Unfortunately, reloptions don't have an > assign_hook like GUCs do. Unless we add those we've no way to still > accept lower values without an error. Does anyone

Re: Plain strdup() in frontend code

2019-04-29 Thread Michael Paquier
On Mon, Apr 29, 2019 at 01:35:12PM +, Daniel Gustafsson wrote: > Good point, I've updated the patch to include those as well. I have been reviewing this patch, and the change in pg_waldump is actually a good thing, as we could finish with a crash if strdup() returns NULL as the pointer gets

Re: Pluggable Storage - Andres's take

2019-04-29 Thread Ashwin Agrawal
On Thu, Apr 25, 2019 at 3:43 PM Andres Freund wrote: > Hm. I think some of those changes would be a bit bigger than I initially > though. Attached is a more minimal fix that'd route > RelationGetNumberOfBlocksForFork() through tableam if necessary. I > think it's definitely the right answer for

Re: jsonpath

2019-04-29 Thread Alexander Korotkov
On Mon, Apr 29, 2019 at 6:11 PM Tom Lane wrote: > Alexander Korotkov writes: > > [ jsonpath-errors-improve-3.patch ] > > This is getting better, but IMO it's still a bit too willing to use > a boilerplate primary error message plus errdetail. I do not think > that is project style nor something

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-04-29 Thread Andres Freund
Hi, On 2019-04-29 18:07:07 -0400, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> Taking this as a WIP, what do you think? > > > Seems generally about right. > > Andres, are you pushing this forward? Next week's minor releases > are coming up fast, and we're going to need to adapt

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-04-29 Thread Tom Lane
I wrote: > Andres Freund writes: >> Taking this as a WIP, what do you think? > Seems generally about right. Andres, are you pushing this forward? Next week's minor releases are coming up fast, and we're going to need to adapt the HEAD patch significantly for the back branches AFAICS. So

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread David Rowley
On Tue, 30 Apr 2019 at 06:28, Peter Geoghegan wrote: > > On Mon, Apr 29, 2019 at 11:20 AM Alvaro Herrera > wrote: > > Agreed. Here's a patch. I see downthread that you also discovered the > > same mistake in _h_indexbuild by grepping for "long"; I got to it by > > examining callers of

Re: speeding up planning with partitions

2019-04-29 Thread Tom Lane
Amit Langote writes: > Here is the patch. I've also included the patch to update the text in > ddl.sgml regarding constraint exclusion and partition pruning. I thought this was a bit messy. In particular, IMV the reason to have a split between get_relation_constraints and its only caller

Re: [PATCH v5] Show detailed table persistence in \dt+

2019-04-29 Thread Fabien COELHO
Hello David, Patch v7 applies, compiles, make check ok. No docs needed. No tests, pending some TAP infrastructure. I could no test with a version between 8.4 & 9.1. No further comments. Marked as ready. -- Fabien.

Re: [PATCH v4] Add \warn to psql

2019-04-29 Thread Fabien COELHO
Hello David, About v5: applies, compiles, global & local make check ok, doc gen ok. Very minor comment: \qecho is just before \o in the embedded help, where it should be just after. Sorry I did not see it on the preceding submission. Done. Patch v6 applies, compiles, global & local make

Re: Calling PrepareTempTablespaces in BufFileCreateTemp

2019-04-29 Thread Peter Geoghegan
On Mon, Apr 29, 2019 at 12:31 PM Ashwin Agrawal wrote: > Well the one thing I wish to point out explicitly is just taking fd.c > changes from [1], and running make check hits no assertions and > doesn't flag issue exist for gistbuildbuffers.c. Means its missing > coverage and in future same can

Re: Calling PrepareTempTablespaces in BufFileCreateTemp

2019-04-29 Thread Ashwin Agrawal
On Thu, Apr 25, 2019 at 11:53 PM Michael Paquier wrote: > > On Thu, Apr 25, 2019 at 12:45:03PM -0400, Tom Lane wrote: > > I still remain concerned that invoking catalog lookups from fd.c is a darn > > bad idea, even if we have a fallback for it to work (for some value of > > "work") in

Re: generate documentation keywords table automatically

2019-04-29 Thread Chapman Flack
On 4/29/19 2:45 PM, Peter Eisentraut wrote: >> A policy issue, independent of this mechanism, is how many different >> SQL spec versions we want to show in the table. > > We had previously established that we want to show 92 and the latest > two. I don't propose to change that. An annoying API

Re: generate documentation keywords table automatically

2019-04-29 Thread Peter Eisentraut
On 2019-04-27 17:25, Tom Lane wrote: > Would it make more sense to have just one source file per SQL standard > version, and distinguish the keyword types by labels within the file? The way I have written it, the lists can be compared directly with the relevant standards by a human. Otherwise

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Peter Geoghegan
On Mon, Apr 29, 2019 at 11:24 AM Andres Freund wrote: > > I don't think that anybody cares about Win64 very much. > > I seriously doubt this assertion. Note that the postgres packages on > https://www.postgresql.org/download/windows/ do not support 32bit > windows anymore (edb from 11 onwards,

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Peter Geoghegan
On Mon, Apr 29, 2019 at 11:20 AM Alvaro Herrera wrote: > Agreed. Here's a patch. I see downthread that you also discovered the > same mistake in _h_indexbuild by grepping for "long"; I got to it by > examining callers of pgstat_progress_update_param and > pgstat_progress_update_multi_param. I

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Andres Freund
Hi, On 2019-04-29 11:18:49 -0700, Peter Geoghegan wrote: > On Mon, Apr 29, 2019 at 11:10 AM Tom Lane wrote: > > If we don't want to rely on "L" constants then we'll have to write these > > cases like "work_mem * (size_t) 1024" which is ugly, lots more keystrokes, > > and prone to weird

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Alvaro Herrera
On 2019-Apr-28, Peter Geoghegan wrote: > Commit ab0dfc961b6 used a "long" variable within _bt_load() to count > the number of tuples entered into a B-Tree index as it is built. This > will not work as expected on Windows, even on 64-bit Windows, because > "long" is only 32-bits wide. It's far

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Peter Geoghegan
On Mon, Apr 29, 2019 at 11:10 AM Tom Lane wrote: > If we don't want to rely on "L" constants then we'll have to write these > cases like "work_mem * (size_t) 1024" which is ugly, lots more keystrokes, > and prone to weird precedence problems unless you throw even more > keystrokes (parentheses)

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Tom Lane
Andres Freund writes: > On 2019-04-29 13:32:13 -0400, Tom Lane wrote: >> There's more to that than you might realize. For example, guc.c >> enforces a limit on work_mem that's designed to ensure that >> expressions like "work_mem * 1024L" won't overflow, and there are >> similar choices

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Peter Geoghegan
On Mon, Apr 29, 2019 at 10:32 AM Tom Lane wrote: > There's more to that than you might realize. For example, guc.c > enforces a limit on work_mem that's designed to ensure that > expressions like "work_mem * 1024L" won't overflow, and there are > similar choices elsewhere. I was aware of that,

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Andres Freund
Hi, On 2019-04-29 13:32:13 -0400, Tom Lane wrote: > There's more to that than you might realize. For example, guc.c > enforces a limit on work_mem that's designed to ensure that > expressions like "work_mem * 1024L" won't overflow, and there are > similar choices elsewhere. I'm not sure if we

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Tom Lane
Ashwin Agrawal writes: > On Mon, Apr 29, 2019 at 10:36 AM Tom Lane wrote: >> Can you try applying a1a789eb5ac894b4ca4b7742f2dc2d9602116e46 >> to see if it fixes the problem for you? > Yes, will give it a try on greenplum and report back the result. > Have we decided if this will be applied to

Re: CHAR vs NVARCHAR vs TEXT performance

2019-04-29 Thread Tom Lane
Rob writes: > Basically, if a table exists with a PK which is CHAR(n) and a query is > sent with VARCHAR or CHAR then it uses an Index Scan. If the query is > sent with TEXT as the type then postgresql casts the column to TEXT > (rather than the value to CHAR) and it does a Seq Scan. Yeah, this

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Ashwin Agrawal
On Mon, Apr 29, 2019 at 10:36 AM Tom Lane wrote: > > Can you try applying a1a789eb5ac894b4ca4b7742f2dc2d9602116e46 > to see if it fixes the problem for you? Yes, will give it a try on greenplum and report back the result. Have we decided if this will be applied to back branches?

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Tom Lane
Ashwin Agrawal writes: > For Greenplum (based on 9.4 but current master code looks the same) we > did see deadlocks recently hit in CI many times for walreceiver which > I believe confirms above finding. > #0 __lll_lock_wait_private () at > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 >

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Tom Lane
Peter Geoghegan writes: > On Mon, Apr 29, 2019 at 8:11 AM Andres Freund wrote: >> I think we should start by just removing all uses of long. There's >> really no excuse for them today, and a lot of them are bugs waiting to >> happen. > I like the idea of banning "long" altogether. It will

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Andres Freund
Hi, On 2019-04-29 10:16:39 -0700, Peter Geoghegan wrote: > On Mon, Apr 29, 2019 at 8:11 AM Andres Freund wrote: > > I think we should start by just removing all uses of long. There's > > really no excuse for them today, and a lot of them are bugs waiting to > > happen. > > I like the idea of

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Ashwin Agrawal
On Sat, Apr 27, 2019 at 5:57 PM Tom Lane wrote: > > I have spent a fair amount of time trying to replicate these failures > locally, with little success. I now think that the most promising theory > is Munro's idea in [1] that the walreceiver is hanging up during its > unsafe attempt to do

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Peter Geoghegan
On Mon, Apr 29, 2019 at 8:11 AM Andres Freund wrote: > I think we should start by just removing all uses of long. There's > really no excuse for them today, and a lot of them are bugs waiting to > happen. I like the idea of banning "long" altogether. It will probably be hard to keep it out of

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Andres Freund
Hi, On 2019-04-29 12:55:31 -0400, Tom Lane wrote: > Andres Freund writes: > > Hm, I'm not convinced that's OK. What if there's a network hickup? We'll > > wait until there's an OS tcp timeout, no? > > No. send() is only going to block if there's no room in the kernel's > buffers, and that

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Tom Lane
Andres Freund writes: > On 2019-04-27 20:56:51 -0400, Tom Lane wrote: >> The existing code does not use PQsetnonblocking, which means that it's >> theoretically at risk of blocking while pushing out data to the remote >> server. In practice I think that risk is negligible because (IIUC) we >>

CHAR vs NVARCHAR vs TEXT performance

2019-04-29 Thread Rob
Hi Guys, I wanted to get some thoughts about a type-specific performance problem we hit through our application tier. The full conversation is here: https://github.com/npgsql/npgsql/issues/2283 Basically, if a table exists with a PK which is CHAR(n) and a query is sent with VARCHAR or

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Andres Freund
Hi, On 2019-04-27 20:56:51 -0400, Tom Lane wrote: > Even if that isn't the proximate cause of the current reports, it's > clearly trouble waiting to happen, and we should get rid of it. > Accordingly, see attached proposed patch. This just flushes the > "immediate interrupt" stuff in favor of

Re: Identity columns should own only one sequence

2019-04-29 Thread Laurenz Albe
On Sat, 2019-04-27 at 14:16 +0200, Peter Eisentraut wrote: > On 2019-04-26 15:37, Laurenz Albe wrote: > > What do you think of the patch I just posted on this thread to > > remove ownership automatically when the default is dropped, as Michael > > suggested? I think that would make things much

Re: Race conditions with checkpointer and shutdown

2019-04-29 Thread Tom Lane
Thomas Munro writes: > On Sun, Apr 28, 2019 at 12:56 PM Tom Lane wrote: >> Even if that isn't the proximate cause of the current reports, it's >> clearly trouble waiting to happen, and we should get rid of it. > +1 > That reminds me, we should probably also clean up at least the >

Re: "long" type is not appropriate for counting tuples

2019-04-29 Thread Andres Freund
Hi, On 2019-04-28 19:24:59 -0400, Tom Lane wrote: > Peter Geoghegan writes: > > ISTM that we should try to come up with a way of making code like this > > work, rather than placing the burden on new code to get it right. > > Other than "use the right datatype", I'm not sure what we can do? > In

Re: jsonpath

2019-04-29 Thread Tom Lane
Alexander Korotkov writes: > [ jsonpath-errors-improve-3.patch ] This is getting better, but IMO it's still a bit too willing to use a boilerplate primary error message plus errdetail. I do not think that is project style nor something to be encouraged. In particular, you've got a whole lot of

Re: pg_ssl

2019-04-29 Thread Stephen Frost
Greetings, * Steve (stev...@osfda.org) wrote: > As you might know, generating SSL certificates for postgres (to be used by > pgadmin, for example...) can be quite a bear; especially if you need more > than one, since they are based on the username of the postgres user. Well, you can map the

Re: Typofixes in src/bin

2019-04-29 Thread Michael Paquier
On Mon, Apr 29, 2019 at 01:40:26PM +, Daniel Gustafsson wrote: > Two random typos spotted while perusing code in src/bin. Thanks, fixed. -- Michael signature.asc Description: PGP signature

Re: [PATCH v5] Show detailed table persistence in \dt+

2019-04-29 Thread David Fetter
On Mon, Apr 29, 2019 at 08:48:17AM +0200, Fabien COELHO wrote: > > Hello David, > > > My mistake. Fixed. > > About v6: applies, compiles, make check ok. > > Code is ok. > > Maybe there could be a comment to tell that prior version are not addressed, > something like: > > ... > } > /*

Re: [PATCH v4] Add \warn to psql

2019-04-29 Thread David Fetter
On Mon, Apr 29, 2019 at 08:30:18AM +0200, Fabien COELHO wrote: > > Hello David, > > About v5: applies, compiles, global & local make check ok, doc gen ok. > > Very minor comment: \qecho is just before \o in the embedded help, where it > should be just after. Sorry I did not see it on the

Typofixes in src/bin

2019-04-29 Thread Daniel Gustafsson
Two random typos spotted while perusing code in src/bin. cheers ./daniel typo-src_bin.patch Description: Binary data

Re: Plain strdup() in frontend code

2019-04-29 Thread Daniel Gustafsson
On Monday, April 29, 2019 3:01 PM, Michael Paquier wrote: > On Mon, Apr 29, 2019 at 11:47:27AM +, Daniel Gustafsson wrote: > > > Reading code I noticed that we in a few rare instances use strdup() in > > frontend > > utilities instead of pg_strdup(). Is there a reason for not using > >

Re: Failure in contrib test _int on loach

2019-04-29 Thread Anastasia Lubennikova
27.04.2019 22:05, Tom Lane wrote: Anastasia Lubennikova writes: So it is possible, but it doesn't require any extra algorithm changes. I didn't manage to generate dataset to reproduce grandparent split. Though, I do agree that it's worth checking out. Do you have any ideas? Ping? This thread

Re: Refactoring the checkpointer's fsync request queue

2019-04-29 Thread Alvaro Herrera
On 2019-Apr-05, Thomas Munro wrote: > Ok, here is a patch that adds a one-typedef header and uses > SegmentIndex to replace all cases of BlockNumber and int holding a > segment number (where as an "index" or a "count"). Hmm, I now see (while doing the pg_checksum translation) that this patch

Re: Plain strdup() in frontend code

2019-04-29 Thread Michael Paquier
On Mon, Apr 29, 2019 at 11:47:27AM +, Daniel Gustafsson wrote: > Reading code I noticed that we in a few rare instances use strdup() in > frontend > utilities instead of pg_strdup(). Is there a reason for not using pg_strdup() > consistently as per the attached patch? I think that it is

Plain strdup() in frontend code

2019-04-29 Thread Daniel Gustafsson
Reading code I noticed that we in a few rare instances use strdup() in frontend utilities instead of pg_strdup(). Is there a reason for not using pg_strdup() consistently as per the attached patch? cheers ./daniel frontend_strdup.patch Description: Binary data

Re: Cleanup/remove/update references to OID column

2019-04-29 Thread Justin Pryzby
On Wed, Apr 17, 2019 at 11:14:13PM -0500, Justin Pryzby wrote: > On Wed, Apr 17, 2019 at 05:51:15PM -0700, Andres Freund wrote: > > > - rows was exactly one and the target table > > > was > > > + count was exactly one and the target table > > > was > > The rows reference is from your change

Re: BUG #15708: RLS 'using' running as wrong user when called from a view

2019-04-29 Thread Dean Rasheed
On Mon, 29 Apr 2019 at 04:56, Daurnimator wrote: > > On Wed, 27 Mar 2019 at 23:46, Dean Rasheed wrote: > > On second thoughts, it actually needs to be in > > get_row_security_policies(), after making copies of the quals from the > > policies, otherwise it would be scribbling on the copies from

Re: Speed up build on Windows by generating symbol definition in batch

2019-04-29 Thread David Rowley
On Mon, 29 Apr 2019 at 14:50, Noah Misch wrote: > > On Wed, Apr 10, 2019 at 02:27:26PM +0800, Peifeng Qiu wrote: > > I've updated the patch according to your comments. > > Looks good. Thanks. I plan to push this on Saturday. I didn't really look at the patch in detail, but on testing it on a

Re: [PATCH v5] Show detailed table persistence in \dt+

2019-04-29 Thread Fabien COELHO
Hello David, My mistake. Fixed. About v6: applies, compiles, make check ok. Code is ok. Maybe there could be a comment to tell that prior version are not addressed, something like: ... } /* else do not bother guessing the temporary status on old version */ No tests, pending an

Re: [PATCH v4] Add \warn to psql

2019-04-29 Thread Fabien COELHO
Hello David, About v5: applies, compiles, global & local make check ok, doc gen ok. Very minor comment: \qecho is just before \o in the embedded help, where it should be just after. Sorry I did not see it on the preceding submission. -- Fabien.