Re: speeding up planning with partitions

2019-03-30 Thread Amit Langote
On Sun, Mar 31, 2019 at 11:45 AM Imai Yoshikazu wrote: > On 2019/03/31 1:06, Amit Langote wrote: > > On Sun, Mar 31, 2019 at 12:11 AM Tom Lane wrote: > >> I am curious as to why there seems to be more degradation > >> for hash cases, as per Yoshikazu-san's results in > >>

Re: [HACKERS] generated columns

2019-03-30 Thread Erik Rijkers
On 2019-01-16 22:40, Erik Rijkers wrote: If you add a generated column to a file_fdw foreign table, it works OK wih VIRTUAL (the default) but with STORED it adds an empty column, silently. I would say it would make more sense to get an error. VIRTUAL is gone, but that other issue is still

Re: Psql patch to show access methods info

2019-03-30 Thread s . cherkashin
Thanks for review. With + it shows description: # \dA+ List of access methods Name | Type | Handler| Description +---+--+--- - brin | index | brinhandler |

Re: Why does ExecComputeStoredGenerated() form a heap tuple

2019-03-30 Thread Andres Freund
Hi, On 2019-03-30 19:57:44 -0700, Andres Freund wrote: > while rebasing the remaining tableam patches (luckily a pretty small set > now!), I had a few conflicts with ExecComputeStoredGenerated(). While > resolving I noticed: > > oldtuple = ExecFetchSlotHeapTuple(slot, true, _free); >

Why does ExecComputeStoredGenerated() form a heap tuple

2019-03-30 Thread Andres Freund
Hi, while rebasing the remaining tableam patches (luckily a pretty small set now!), I had a few conflicts with ExecComputeStoredGenerated(). While resolving I noticed: oldtuple = ExecFetchSlotHeapTuple(slot, true, _free); newtuple = heap_modify_tuple(oldtuple, tupdesc, values,

Re: speeding up planning with partitions

2019-03-30 Thread Imai Yoshikazu
On 2019/03/31 1:06, Amit Langote wrote: > On Sun, Mar 31, 2019 at 12:11 AM Tom Lane wrote: >> Amit Langote writes: >>> I think the performance results did prove that degradation due to >>> those loops over part_rels becomes significant for very large >>> partition counts. Is there a better

Re: dropdb --force

2019-03-30 Thread Andres Freund
Hi, On 2019-03-10 11:20:42 +0100, Filip Rembiałkowski wrote: > bool > -CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared) > +CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared, bool > force_terminate) > { That doesn't seem like a decent API to me.

Re: dropdb --force

2019-03-30 Thread Ryan Lambert
Hello, This is a feature I have wanted for a long time, thank you for your work on this. The latest patch [1] applied cleanly for me. In dbcommands.c the comment references a 5 second delay, I don't see where that happens, am I missing something? I tested both the dropdb program and the in

Re: patch to allow disable of WAL recycling

2019-03-30 Thread Tomas Vondra
On Fri, Mar 29, 2019 at 01:09:46PM +1300, Thomas Munro wrote: ... I still don't know why exactly this happens, but it's clearly a real phenomenon. As for why Tomas Vondra couldn't see it, I'm guessing that stacks more RAM and ~500k IOPS help a lot (essentially the opposite end of the memory,

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-30 Thread Tomas Vondra
On Sun, Mar 31, 2019 at 08:50:53AM +0800, John Naylor wrote: I believe I found a typo in mcv.c, fix attached. Thanks, pushed. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-30 Thread John Naylor
I believe I found a typo in mcv.c, fix attached. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services mcv-comment-fix.patch Description: Binary data

Re: Re: FETCH FIRST clause WITH TIES option

2019-03-30 Thread Tomas Vondra
On Sun, Mar 31, 2019 at 01:14:46AM +0100, Tomas Vondra wrote: On Fri, Mar 29, 2019 at 01:56:48AM +0100, Tomas Vondra wrote: On Tue, Mar 26, 2019 at 10:46:00AM +0300, Surafel Temesgen wrote: On Mon, Mar 25, 2019 at 11:56 AM David Steele wrote: This patch no longer passes testing so marked

Re: Re: FETCH FIRST clause WITH TIES option

2019-03-30 Thread Tomas Vondra
On Fri, Mar 29, 2019 at 01:56:48AM +0100, Tomas Vondra wrote: On Tue, Mar 26, 2019 at 10:46:00AM +0300, Surafel Temesgen wrote: On Mon, Mar 25, 2019 at 11:56 AM David Steele wrote: This patch no longer passes testing so marked Waiting on Author. Thank you for informing. Fixed Thanks

clean up docs for v12

2019-03-30 Thread Justin Pryzby
I reviewed docs like this: git log -p remotes/origin/REL_11_STABLE..HEAD -- doc And split some into separate patches, which may be useful at least for reviewing. I'm mailing now rather than after feature freeze to avoid duplicative work and see if there's any issue. Note, I also/already mailed

Re: Teach pg_upgrade test to honor NO_TEMP_INSTALL

2019-03-30 Thread Tom Lane
Andrew Dunstan writes: > On some machines (*cough* Mingw *cough*) installs are very slow. We've > ameliorated this by allowing temp installs to be reused, but the > pg_upgrade Makefile never got the message. Here's a patch that does > that. I'd like to backpatch it, at least to 9.5 where we

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Andres Freund
Hi, On March 30, 2019 5:33:12 PM EDT, Thomas Munro wrote: >On Sun, Mar 31, 2019 at 8:20 AM Peter Geoghegan wrote: >> On Sat, Mar 30, 2019 at 8:44 AM Robert Haas >wrote: >> > Overall I'm inclined to think that we're making the same mistake >here >> > that we did with work_mem, namely, assuming

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Thomas Munro
On Sun, Mar 31, 2019 at 8:20 AM Peter Geoghegan wrote: > On Sat, Mar 30, 2019 at 8:44 AM Robert Haas wrote: > > Overall I'm inclined to think that we're making the same mistake here > > that we did with work_mem, namely, assuming that you can control a > > bunch of different prefetching

Re: Teach pg_upgrade test to honor NO_TEMP_INSTALL

2019-03-30 Thread Daniel Gustafsson
On Saturday, March 30, 2019 9:42 PM, Andrew Dunstan wrote: > On some machines (cough Mingw cough) installs are very slow. We've > ameliorated this by allowing temp installs to be reused, but the > pg_upgrade Makefile never got the message. Here's a patch that does > that. I'd like to backpatch

Re: Compressed TOAST Slicing

2019-03-30 Thread Stephen Frost
Greetings, * Paul Ramsey (pram...@cleverelephant.ca) wrote: > > On Mar 19, 2019, at 4:47 AM, Stephen Frost wrote: > > * Paul Ramsey (pram...@cleverelephant.ca) wrote: > >>> On Mar 18, 2019, at 7:34 AM, Robert Haas wrote: > >>> +1. I think Paul had it right originally. > >> > >> In that

Teach pg_upgrade test to honor NO_TEMP_INSTALL

2019-03-30 Thread Andrew Dunstan
On some machines (*cough* Mingw *cough*) installs are very slow. We've ameliorated this by allowing temp installs to be reused, but the pg_upgrade Makefile never got the message. Here's a patch that does that. I'd like to backpatch it, at least to 9.5 where we switched the pg_upgrade location.

Re: Enable data checksums by default

2019-03-30 Thread Andres Freund
On March 30, 2019 3:25:43 PM EDT, Tomas Vondra wrote: >On Fri, Mar 29, 2019 at 08:35:26PM +0100, Christoph Berg wrote: >>Re: Bernd Helmle 2019-03-29 ><3586bb9345a59bfc8d13a50a7c729be1ee6759fd.ca...@oopsware.de> >>> Am Freitag, den 29.03.2019, 23:10 +0900 schrieb Michael Paquier: >>> > >>> > I

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-30 Thread Tomas Vondra
On Wed, Mar 27, 2019 at 08:55:07PM +0100, Tomas Vondra wrote: Hi, I've now committed the MCV part, ... Hmmm, what's the right status in the CF app when a part of a patch was committed and the rest should be moved to the next CF? Committed, Moved to next CF, or something else? regards --

Re: Progress reporting for pg_verify_checksums

2019-03-30 Thread Michael Banck
Hi, so we are basically back at the original patch as written by Bernd :) > +/* > + * Report current progress status. Parts borrowed from > + * PostgreSQLs' src/bin/pg_basebackup.c > + */ > +static void > +progress_report(bool force) > +{ > + int percent; > + char 

Re: Enable data checksums by default

2019-03-30 Thread Tomas Vondra
On Fri, Mar 29, 2019 at 08:35:26PM +0100, Christoph Berg wrote: Re: Bernd Helmle 2019-03-29 <3586bb9345a59bfc8d13a50a7c729be1ee6759fd.ca...@oopsware.de> Am Freitag, den 29.03.2019, 23:10 +0900 schrieb Michael Paquier: > > I can't really believe that many people set up shared_buffers at > 128kB

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Peter Geoghegan
On Sat, Mar 30, 2019 at 8:44 AM Robert Haas wrote: > Overall I'm inclined to think that we're making the same mistake here > that we did with work_mem, namely, assuming that you can control a > bunch of different prefetching behaviors with a single GUC and things > will be OK. Let's just create

Re: Progress reporting for pg_verify_checksums

2019-03-30 Thread Fabien COELHO
Bonjour Michaël, Getting to know the total size and the current size are the two important factors that matter when it comes to do progress reporting in my opinion. I have read the patch, and I am not really convinced by the need to show the progress report based on an interval of 250ms as we

Re: Checksum errors in pg_stat_database

2019-03-30 Thread Julien Rouhaud
Sorry for delay, I had to catch a train. On Sat, Mar 30, 2019 at 4:02 PM Magnus Hagander wrote: > > My vote is still to drop it completely, but if we're keeping it, it has to go > in both paths. Ok. For now I'm attaching v2, which drops this field, rename the view to pg_stat_checksums

Re: Berserk Autovacuum (let's save next Mandrill)

2019-03-30 Thread Vik Fearing
On 27/03/2019 21:54, Darafei "Komяpa" Praliaskouski wrote: > Hi hackers, > > Attached is sketch of small patch that fixes several edge cases with > autovacuum. Long story short autovacuum never comes to append only > tables, killing large productions. > > First case, mine. >   >

Re: speeding up planning with partitions

2019-03-30 Thread Robert Haas
On Sat, Mar 30, 2019 at 12:16 PM Amit Langote wrote: > Fwiw, I had complained when reviewing the run-time pruning patch that > creating those maps in the planner and putting them in > PartitionPruneInfo might not be a good idea, but David insisted that > it'd be good for performance (in the

Re: Berserk Autovacuum (let's save next Mandrill)

2019-03-30 Thread Amit Langote
On Sun, Mar 31, 2019 at 1:11 AM Robert Haas wrote: > > On Wed, Mar 27, 2019 at 5:32 PM Alvaro Herrera > wrote: > > * certain tables would have some sort of partial scan that sets the > > visibility map. There's no reason to invoke the whole vacuuming > > machinery. I don't think this is

Re: speeding up planning with partitions

2019-03-30 Thread Amit Langote
On Sun, Mar 31, 2019 at 12:59 AM Robert Haas wrote: > > On Sat, Mar 30, 2019 at 11:46 AM Tom Lane wrote: > > > The only problem with PartitionPruneInfo structures of which I am > > > aware is that they rely on PartitionDesc offsets not changing. But I > > > added code in that commit in

Re: Berserk Autovacuum (let's save next Mandrill)

2019-03-30 Thread Robert Haas
On Wed, Mar 27, 2019 at 5:32 PM Alvaro Herrera wrote: > * certain tables would have some sort of partial scan that sets the > visibility map. There's no reason to invoke the whole vacuuming > machinery. I don't think this is limited to append-only tables, but > rather those are just the

Re: speeding up planning with partitions

2019-03-30 Thread Amit Langote
On Sun, Mar 31, 2019 at 12:11 AM Tom Lane wrote: > Amit Langote writes: > > I think the performance results did prove that degradation due to > > those loops over part_rels becomes significant for very large > > partition counts. Is there a better solution than the bitmapset that > > you have

Re: speeding up planning with partitions

2019-03-30 Thread Robert Haas
On Sat, Mar 30, 2019 at 11:46 AM Tom Lane wrote: > > The only problem with PartitionPruneInfo structures of which I am > > aware is that they rely on PartitionDesc offsets not changing. But I > > added code in that commit in ExecCreatePartitionPruneState to handle > > that exact problem. See

Re: speeding up planning with partitions

2019-03-30 Thread Tom Lane
Robert Haas writes: > On Sat, Mar 30, 2019 at 11:11 AM Tom Lane wrote: >> Before that, though, I remain concerned that the PartitionPruneInfo >> data structure the planner is transmitting to the executor is unsafe >> against concurrent ATTACH PARTITION operations. The comment for >>

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Robert Haas
On Sat, Mar 30, 2019 at 6:33 AM Thomas Munro wrote: > I didn't understand that last sentence. > > Here's an attempt to write a suitable comment for the quick fix. And > I suppose effective_io_concurrency is a reasonable default. > > It's pretty hard to think of a good way to get your hands on

Re: speeding up planning with partitions

2019-03-30 Thread Robert Haas
On Sat, Mar 30, 2019 at 11:11 AM Tom Lane wrote: > Before that, though, I remain concerned that the PartitionPruneInfo > data structure the planner is transmitting to the executor is unsafe > against concurrent ATTACH PARTITION operations. The comment for > PartitionedRelPruneInfo says in so

Re: jsonpath

2019-03-30 Thread Alexander Korotkov
On Fri, Mar 29, 2019 at 4:15 PM Alexander Korotkov wrote: > On Thu, Mar 28, 2019 at 7:43 PM Andrew Dunstan > wrote: > > On 3/28/19 9:50 AM, Tom Lane wrote: > > > Andres Freund writes: > > >> On March 28, 2019 9:31:14 AM EDT, Tom Lane wrote: > > >>> Has anybody gotten through a valgrind run on

Re: speeding up planning with partitions

2019-03-30 Thread Tom Lane
Amit Langote writes: > On Sat, Mar 30, 2019 at 9:17 AM Tom Lane wrote: >> What I propose we do about the GEQO problem is shown in 0001 attached >> (which would need to be back-patched into v11). >> ... >> That's just dumb. What we *ought* to be doing in such degenerate >> outer-join cases is

Re: Checksum errors in pg_stat_database

2019-03-30 Thread Magnus Hagander
On Sat, Mar 30, 2019 at 3:55 PM Julien Rouhaud wrote: > On Sat, Mar 30, 2019 at 2:33 PM Magnus Hagander > wrote: > > > > On Wed, Mar 13, 2019 at 4:54 PM Julien Rouhaud > wrote: > >> > >> On Wed, Mar 13, 2019 at 4:53 PM Julien Rouhaud > wrote: > >> > > >> > As a result I ended up simply adding

Re: Checksum errors in pg_stat_database

2019-03-30 Thread Julien Rouhaud
On Sat, Mar 30, 2019 at 2:33 PM Magnus Hagander wrote: > > On Wed, Mar 13, 2019 at 4:54 PM Julien Rouhaud wrote: >> >> On Wed, Mar 13, 2019 at 4:53 PM Julien Rouhaud wrote: >> > >> > As a result I ended up simply adding counters for the number of total >> > checks and the timestamp of the last

Re: Progress reporting for pg_verify_checksums

2019-03-30 Thread Michael Paquier
On Thu, Mar 28, 2019 at 03:53:59PM +0100, Fabien COELHO wrote: > I think that it is good to show the overall impact of the signal stuff, in > particular the fact that the size must always be computed if the progress > may be activated. Getting to know the total size and the current size are the

Re: Column lookup in a row performance

2019-03-30 Thread Tom Lane
=?UTF-8?B?0J/QsNCy0LvRg9GF0LjQvSDQmNCy0LDQvQ==?= writes: > Does anyone know why the format is still the same? (1) Backwards compatibility, and (2) it's not clear that a different layout would be a win for all cases. regards, tom lane

[PATCH v22] GSSAPI encryption support

2019-03-30 Thread Stephen Frost
Greetings, * Robbie Harwood (rharw...@redhat.com) wrote: > Stephen Frost writes: > > > One of the things that I really didn't care for in this patch was the > > use of the string buffers, without any real checks (except for "oh, > > you tried to allocated over 1G"...) to make sure that the

Re: Indexscan failed assert caused by using index without lock

2019-03-30 Thread Tom Lane
=?UTF-8?B?6auY5aKe55Cm?= writes: > Following example can reproduce the problem: Yeah, this is being discussed at https://www.postgresql.org/message-id/flat/19465.1541636...@sss.pgh.pa.us regards, tom lane

Re: Online verification of checksums

2019-03-30 Thread Andres Freund
Hi, On 2019-03-30 12:56:21 +0100, Magnus Hagander wrote: > > ISTM that the fact that we had to teach it about different segment files > > for checksum verification by splitting up the filename at "." implies > > that it is not the correct level of abstraction (but maybe it could get > > schooled

Re: Checksum errors in pg_stat_database

2019-03-30 Thread Magnus Hagander
On Wed, Mar 13, 2019 at 4:54 PM Julien Rouhaud wrote: > On Wed, Mar 13, 2019 at 4:53 PM Julien Rouhaud wrote: > > > > On Sun, Mar 10, 2019 at 1:13 PM Julien Rouhaud > wrote: > > > > > > On Sat, Mar 9, 2019 at 7:58 PM Julien Rouhaud > wrote: > > > > > > > > On Sat, Mar 9, 2019 at 7:50 PM

Re: pgsql: Improve autovacuum logging for aggressive and anti-wraparound ru

2019-03-30 Thread Andrew Dunstan
On 3/29/19 9:08 PM, Michael Paquier wrote: > On Fri, Mar 29, 2019 at 11:22:55AM -0300, Alvaro Herrera wrote: >> Yeah, that looks good to me too. I wonder if we really need it as LOG >> though; we don't say anything for actions unless they take more than the >> min duration, so why say something

Re: Online verification of checksums

2019-03-30 Thread Magnus Hagander
On Fri, Mar 29, 2019 at 10:08 PM Michael Banck wrote: > Hi, > > Am Freitag, den 29.03.2019, 16:52 +0100 schrieb Magnus Hagander: > > On Fri, Mar 29, 2019 at 4:30 PM Stephen Frost > wrote: > > > * Magnus Hagander (mag...@hagander.net) wrote: > > > > On Thu, Mar 28, 2019 at 10:19 PM Tomas Vondra

Re: PostgreSQL pollutes the file system

2019-03-30 Thread Peter Eisentraut
On 2019-03-29 20:32, Joe Conway wrote: > pg_util How is that better than just renaming to pg_$oldname? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PostgreSQL pollutes the file system

2019-03-30 Thread Peter Eisentraut
On 2019-03-29 16:41, Tom Lane wrote: > Or perhaps better, allow pg_ctl to grow new > subcommands for those tasks? pg_ctl is a tool to control the server; the commands being complained about are client-side things. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL

Re: REINDEX CONCURRENTLY 2.0

2019-03-30 Thread Peter Eisentraut
On 2019-03-29 16:10, Shinoda, Noriyoshi (PN Japan A Delivery) wrote: > postgres=> CREATE TABLE part1(c1 INT) PARTITION BY RANGE(c1); > CREATE TABLE > postgres=> CREATE TABLE part1v1 PARTITION OF part1 FOR VALUES FROM (0) TO > (100); > CREATE TABLE > postgres=> CREATE INDEX idx1_part1 ON

Re: PostgreSQL pollutes the file system

2019-03-30 Thread Fred .Flintstone
I think the proposal you put forward is great, and would love to see it go ahead and get implemented. On Fri, Mar 29, 2019 at 5:35 PM Alvaro Herrera wrote: > > On 2019-Mar-29, Tom Lane wrote: > > > Christoph Berg writes: > > > What might possibly make sense is to add options to psql to > > >

Re: Unix socket dir, an idea

2019-03-30 Thread Danylo Hlynskyi
Hi Tom, and much thanks for reply! > I would also like to point out the extreme Unix-centricity (and > even particular-distribution-centricity) of the alternative locations > you mention Yes! The /run/user and /var/run directories are absent on MacOS. That's why I **don't** propose to change

PostgreSQL 12 Release Management Team & Feature Freeze

2019-03-30 Thread Michael Paquier
Hi, The Release Management Team (RMT) for the PostgreSQL 12 release has been assembled and has determined that the feature freeze date for the PostgreSQL 12 release will be April 7, 2019. This means that any feature that will be going into the PostgreSQL 12 release must be committed before

Re: [HACKERS] generated columns

2019-03-30 Thread Justin Pryzby
On Sat, Mar 30, 2019 at 09:03:03AM +0100, Peter Eisentraut wrote: > On 2019-03-26 20:50, Pavel Stehule wrote: > > It is great feature and I'll mark this feature as ready for commit > > Committed, thanks. create_table.sgml now has this:

Removing a few more lseek() calls

2019-03-30 Thread Thomas Munro
Hello, Patch 0001 gets rid of the unconditional lseek() calls for SLRU I/O, as a small follow-up to commit c24dcd0c. Patch 0002 gets rid of a few places that usually do a good job of avoiding lseek() calls while reading and writing WAL, but it seems better to have no code at all. -- Thomas

Re: [HACKERS] generated columns

2019-03-30 Thread Pavel Stehule
so 30. 3. 2019 v 9:03 odesílatel Peter Eisentraut < peter.eisentr...@2ndquadrant.com> napsal: > On 2019-03-26 20:50, Pavel Stehule wrote: > > It is great feature and I'll mark this feature as ready for commit > > Committed, thanks. > great feature, it reduce some necessity of triggers Regards

Re: [HACKERS] generated columns

2019-03-30 Thread Peter Eisentraut
On 2019-03-26 20:50, Pavel Stehule wrote: > It is great feature and I'll mark this feature as ready for commit Committed, thanks. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Column lookup in a row performance

2019-03-30 Thread Павлухин Иван
Hi PostgresSQL developers, I asked my question already on pgsql-general list and did not find an explanation. Below is the question mainly copied from [0]. I am learning deeply how tuples are organized and column values are accessed in different databases. As far as undertood postgres does

Indexscan failed assert caused by using index without lock

2019-03-30 Thread 高增琦
Following example can reproduce the problem: ``` create table d(a int); create index di on d(a); set enable_seqscan=off; set enable_bitmapscan to off; prepare p as delete from d where a=3; execute p; execute p; ``` The reason is that: ExecInitIndexScan will not lock index because it thinks