[PERFORM] query - laziness of lateral join with function

2015-02-12 Thread paulcc
Hi I'm using cross join lateral with a non-trivial function in an attempt to limit calculation of that function, and am wondering about some aspects of how lateral is currently implemented. NB these queries are generated by a certain ORM, and are usually embedded in much more complex

Re: [PERFORM] Survey: Max TPS you've ever seen

2015-02-12 Thread Gudmundsson Martin (mg)
Hi all! - checkpoint_segments 1000 - checkpoint_completion_target 0.9 - wal_buffers  256MB - shared_buffers 31 gb - max_connections 500 I see that some of you are using wal_buffers = 256MB. I was under the impression that Postgres will not benefit from higher value than the segment size,

Re: [PERFORM] Survey: Max TPS you've ever seen

2015-02-12 Thread Luis Antonio Dias de Sá Junior
For me 12000 tps until now 24 core, 150 Gb ram - 5 ssd raid 5 - Debian 7.8 - Postgres 9.3.5 ...with Postgres parameters customized: - checkpoint_segments 1000 - checkpoint_completion_target 0.9 - wal_buffers 256MB - shared_buffers 31 gb - max_connections 500 - effective_io_concurrency 15

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Kevin Grittner
Nico Sabbi nicola.sa...@poste.it wrote: Queries get executed very very slowly, say 20 minutes. I'd like to know if someone has already succeeded in running postgres with 200-300M records with queries running much faster than this. If you go to the http://wcca.wicourts.gov/ web site, bring

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Nico Sabbi
On 02/12/2015 11:38 PM, Kevin Grittner wrote: If you go to the http://wcca.wicourts.gov/ web site, bring up any case, and click the Court Record Events button, it will search a table with hundreds of millions of rows. The table is not partitioned, but has several indexes on it which are useful

Re: [PERFORM] query - laziness of lateral join with function

2015-02-12 Thread Tom Lane
paulcc paulcc@gmail.com writes: select count(alpha.id) from alpha cross join lateral some_function(alpha.id) as some_val where alpha.test Here the function is strict, and moreover its argument will never be null - hence there should always be a non-null value

[PERFORM] Configuration tips for very large database

2015-02-12 Thread Nico Sabbi
Hello, I've been away from postgres for several years, so please forgive me if I forgot nearly everything:-) I've just inherited a database collecting environmental data. There's a background process continually inserting records (not so often, to say the truth) and a web interface to query

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Kevin Grittner
Nico Sabbi nicola.sa...@poste.it wrote: Can you give any hint on the configuration and on the underlying hardware? Well, this particular web site has millions of hits per day (running up to about 20 queries per hit) from thousands of concurrent web users, while accepting logical replication

Re: [PERFORM] query - laziness of lateral join with function

2015-02-12 Thread David G Johnston
Tom Lane-2 wrote paulcc lt; paulcc.two@ gt; writes: select count(alpha.id) from alpha cross join lateral some_function(alpha.id) as some_val where alpha.test Here the function is strict, and moreover its argument will never be null - hence there should always be a

Re: [PERFORM] query - laziness of lateral join with function

2015-02-12 Thread Paul Callaghan
On Feb 12, 2015 9:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: The planner might produce such a result if there's an opportunity to perform the sorting via an index on alpha (ie, the ORDER BY matches some index). If it has to do an explicit sort it's gonna do the join first. (If you have such

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Mathis, Jason
I can't speak to the numbers postgresql can or cannot do but the numbers above sound very very doable. If you can get a hold of *greg smith's postgresql high performance*, I always liked his method of tuning buffers and checkpoints using the background writer stats. All of which can help with the

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Claudio Freire
On Thu, Feb 12, 2015 at 7:38 PM, Kevin Grittner kgri...@ymail.com wrote: Nico Sabbi nicola.sa...@poste.it wrote: Queries get executed very very slowly, say 20 minutes. I'd like to know if someone has already succeeded in running postgres with 200-300M records with queries running much faster

Re: [PERFORM] Survey: Max TPS you've ever seen

2015-02-12 Thread Mark Kirkwood
On 13/02/15 00:20, Gudmundsson Martin (mg) wrote: Hi all! - checkpoint_segments 1000 - checkpoint_completion_target 0.9 - wal_buffers 256MB - shared_buffers 31 gb - max_connections 500 I see that some of you are using wal_buffers = 256MB. I was under the impression that Postgres will not

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2015-02-12 Thread Michael Paquier
On Wed, Dec 17, 2014 at 4:55 PM, Simon Riggs si...@2ndquadrant.com wrote: On 12 December 2014 at 03:31, Simon Riggs si...@2ndquadrant.com wrote: Also attached is a new parameter called enable_sortedpath which can be used to turn on/off the sorted path generated by the planner. Now with

Re: [PERFORM] Survey: Max TPS you've ever seen

2015-02-12 Thread Graeme B. Bell
1. O/S Under O/S, don't forget to mention linux kernel version. We saw a MASSIVE increase in TPS (I think it was a doubling? Don't have the data to hand right now) on our multicore RHEL6 servers, when moving from a stock RHEL6 kernel to an ELREPO 3.18 series kernel. That's what 10 years of