Re: pg_dump and search_path

2019-07-08 Thread Igal @ Lucee.org
On 7/8/2019 11:48 PM, Igal @ Lucee.org wrote: I have a custom search_path: # show search_path;    search_path --  "staging, transient, pg_catalog" (1 row) I ran `pg_dump --schema-only` and the only reference in the output to search_path is:   SELECT p

pg_dump and search_path

2019-07-08 Thread Igal @ Lucee.org
I have a custom search_path: # show search_path;    search_path --  "staging, transient, pg_catalog" (1 row) I ran `pg_dump --schema-only` and the only reference in the output to search_path is:   SELECT pg_catalog.set_config('search_path', '', false);

Re: OOM with many sorts

2019-07-08 Thread Thomas Munro
On Tue, Jul 9, 2019 at 4:44 AM Justin Pryzby wrote: > [...many partial GroupAggregate/Sort repeated > here for ~200 child tables...] > > This isn't the first time I've had to reduce work_mem on a parallel query to > avoid OOM, but it seems unreasonable to be concerned

Re: pgpool, pgmaster and pgslave migration to ubuntu 18.04

2019-07-08 Thread Tatsuo Ishii
> Hi, > > I'm exploring the options to migrate postgresql master, slave along with > pgpool from ubuntu14.04 to 18.04. Please help me understand the following. > > 1. What are the available options for migrating from one version of O/S to > the other (Here ubuntu 14.04 to 18.04) ? > 2. Document r

pgpool, pgmaster and pgslave migration to ubuntu 18.04

2019-07-08 Thread a venkatesh
Hi, I'm exploring the options to migrate postgresql master, slave along with pgpool from ubuntu14.04 to 18.04. Please help me understand the following. 1. What are the available options for migrating from one version of O/S to the other (Here ubuntu 14.04 to 18.04) ? 2. Document references. 3. An

Re: Elastic Search much faster at statistics?

2019-07-08 Thread Miles Elam
Not enough information. It looks far more like he's testing Ruby's support for ElasticSearch vs ActiveRecord rather than ES vs PostgreSQL. Caching could definitely hold a role but also choice of indexes. If ES is calculating some aggregate info on the fly, the equivalent in PG would be a stats tabl

PGPOOL Question

2019-07-08 Thread Andrew Kerber
I know this is the wrong list for this question, but the pgpool list doesnt appear to be working currently. I am hoping someone on this list can point me to an answer. I have a fairly (I thought) straightforward pgpool setup, the backend is postrgresql database that is streaming to a hot standby

Re: REINDEX : new parameter to preserve current average leaf density as new implicit FILLFACTOR

2019-07-08 Thread Peter Geoghegan
On Mon, Jul 8, 2019 at 12:19 PM Peter Geoghegan wrote: > Well, you're still running autovacuum very aggressively here. It'll > easily keep up when run on a relatively small table such as this. Also, an exactly equal number of insertions and deletions is rather likely to result in bloated indexes

Re: Measuring the Query Optimizer Effect: Turning off the QO?

2019-07-08 Thread Tom Mercha
On 08/07/2019 16:23, Andrew Gierth wrote: >> "Tom" == Tom Mercha writes: > > Tom> Hi All > > Tom> As we know, a query goes through number of stages before it is > Tom> executed. One of these stages is query optimization (QO). > > That's not really true at all. One of the stages is que

Re: REINDEX : new parameter to preserve current average leaf density as new implicit FILLFACTOR

2019-07-08 Thread Peter Geoghegan
On Mon, Jul 8, 2019 at 12:10 PM John Lumby wrote: > Actually the test workload does not run any explicit VACUUM command, > it relies on autovacuum with these settings > (same settings for 9.4 and 12beta2) > To correspond to your " more churn between each VACUUM" > Would you then suggest increasin

Re: Measuring the Query Optimizer Effect: Turning off the QO?

2019-07-08 Thread Tom Mercha
On 08/07/2019 18:29, Tom Lane wrote: > Andrew Gierth writes: >> "Tom" == Tom Lane writes: >> Tom> Two I'd particularly draw your attention to are >> Tom> join_collapse_limit and from_collapse_limit --- if you set both to >> Tom> 1, that'll effectively disable searching for a good join order

Re: REINDEX : new parameter to preserve current average leaf density as new implicit FILLFACTOR

2019-07-08 Thread John Lumby
Thanks Peter > From: Peter Geoghegan > Sent: July 8, 2019 1:39 PM > Subject: Re: REINDEX : new parameter to preserve current average leaf density > as new implicit FILLFACTOR > > Perhaps you didn't take deleted_pages into account -- there must be > free space that is reusable by the index that

Re: REINDEX : new parameter to preserve current average leaf density as new implicit FILLFACTOR

2019-07-08 Thread Peter Geoghegan
On Mon, Jul 8, 2019 at 9:23 AM John Lumby wrote: > Overall, pg-12beta2 yielded a 6.7% reduction in sizes (total pages) of > indexes, which was most noticable with the 6 non-unique ones. > In fact the primary-key index was larger with pg-12. The avg_leaf_density was actually higher for the pri

Elastic Search much faster at statistics?

2019-07-08 Thread Guyren Howe
I find this… surprising. Caching? http://blog.nrowegt.com/database-vs-elasticsearch-speed-column-statistics/

OOM with many sorts

2019-07-08 Thread Justin Pryzby
One of our instances was killed: | Out of memory: Kill process 27670 (postmaster) score 381 or sacrifice child | Killed process 27670 (postmaster) total-vm:13207796kB, anon-rss:5736kB, file-rss:0kB, shmem-rss:12421696kB The VM has: 32GB RAM, shared_buffers=12GB, work_mem=128MB Running only pg11

Re: Measuring the Query Optimizer Effect: Turning off the QO?

2019-07-08 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> Two I'd particularly draw your attention to are > Tom> join_collapse_limit and from_collapse_limit --- if you set both to > Tom> 1, that'll effectively disable searching for a good join order, > Tom> causing the join order to match the

Re: REINDEX : new parameter to preserve current average leaf density as new implicit FILLFACTOR

2019-07-08 Thread John Lumby
> From: Peter Geoghegan > Sent: June 26, 2019 12:09 PM > Subject: Re: REINDEX : new parameter to preserve current average leaf density > as new implicit FILLFACTOR > > > > > > > I suspect that you might find that the enhancements to B-Tree indexes > > > that went into Postgres 12 would help wit

Re: Measuring the Query Optimizer Effect: Turning off the QO?

2019-07-08 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> Two I'd particularly draw your attention to are Tom> join_collapse_limit and from_collapse_limit --- if you set both to Tom> 1, that'll effectively disable searching for a good join order, Tom> causing the join order to match the syntactic structure of the

Re: Measuring the Query Optimizer Effect: Turning off the QO?

2019-07-08 Thread Andrew Gierth
> "Tom" == Tom Mercha writes: Tom> Hi All Tom> As we know, a query goes through number of stages before it is Tom> executed. One of these stages is query optimization (QO). That's not really true at all. One of the stages is query _planning_, which takes the (rewritten) query as input an

Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Tatsuo Ishii
> Thank you for your response, I have followed the discussion on Hackers with > interest. I hope that your efforts are a great success! In my case, I need > to find a solution available in shipping versions of Postgres. But, since > you've joined in, I'm curious: What is the advantage of a material

Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Morris de Oryx
Thanks Steven, nice suggestions. I should have mentioned that the deployment setup is on RDS on PG 11.x, which rules out those extensions. I've looked at TimescaleDB several times, and it looks pretty great. I've now read through some of the archives from years back when pg_xact_commit_timestamp w

RE: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Steven Winfield
There are a couple of extensions that might help you: PipelineDB[1]: Their “Continuous Views” could be useful. A big caveat here is that PipelineDB’s future is uncertain[2], though. I haven’t used it myself, but it looks like you could put triggers onto your existing tables to insert data in Pip

Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Morris de Oryx
Tatsuo, Thank you for your response, I have followed the discussion on Hackers with interest. I hope that your efforts are a great success! In my case, I need to find a solution available in shipping versions of Postgres. But, since you've joined in, I'm curious: What is the advantage of a materia

Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Tatsuo Ishii
> * Full queries will take too long, and will scale poorly. So, MATERIALIZED > VIEW is unappealing. So, rollup tables as it's possible to update them > incrementally. F.Y.I. There is a proposal to implemnt incremental updation against MATERIALIZED VIEW. It is still in WIP patch but currently it su

Incremental aggregate/rollup strategy advice

2019-07-08 Thread Morris de Oryx
I'm researching strategies for incrementally updating aggregate/rollup tables. The problem is how to do so without double-counting changes, and not skipping changes. I know enough about concurrency issues to ask the question, but do *not* know enough about the features and details of Postgres' conc

Re: Active connections are terminated because of small wal_sender_timeout

2019-07-08 Thread Michael Paquier
On Fri, Jul 05, 2019 at 10:03:16AM -0400, Tom Lane wrote: > ayaho...@ibagroup.eu writes: >> Do you have any thoughts regarding this issue? > > I do not think anybody thinks this is a bug. Setting wal_sender_timeout > too small is a configuration mistake. Yeah. I don't see any bug here. Please