Re: [PERFORM] Configuration tips for very large database

2015-02-14 Thread Nico Sabbi
On 02/13/2015 12:19 AM, Claudio Freire wrote: I have a table with ~800M rows, wide ones, that runs reporting queries quite efficiently (usually seconds). Of course, the queries don't traverse the whole table. That wouldn't be efficient. That's probably the key there, don't make you database

Re: [PERFORM] Configuration tips for very large database

2015-02-13 Thread Graeme B. Bell
Hi Nico, No one has mentioned the elephant in the room, but a database can be very I/O intensive and you may not be getting the performance you need from your virtual disk running on your VMware disk subsystem. What do IOmeter or other disk performance evaluation software report?

Re: [PERFORM] Configuration tips for very large database

2015-02-13 Thread k...@rice.edu
On Thu, Feb 12, 2015 at 11:25:54PM +0100, Nico Sabbi wrote: Hello, I've been away from postgres for several years, so please forgive me if I forgot nearly everything:-) I've just inherited a database collecting environmental data. There's a background process continually inserting records

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Kevin Grittner
Nico Sabbi nicola.sa...@poste.it wrote: Queries get executed very very slowly, say 20 minutes. I'd like to know if someone has already succeeded in running postgres with 200-300M records with queries running much faster than this. If you go to the http://wcca.wicourts.gov/ web site, bring

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Nico Sabbi
On 02/12/2015 11:38 PM, Kevin Grittner wrote: If you go to the http://wcca.wicourts.gov/ web site, bring up any case, and click the Court Record Events button, it will search a table with hundreds of millions of rows. The table is not partitioned, but has several indexes on it which are useful

[PERFORM] Configuration tips for very large database

2015-02-12 Thread Nico Sabbi
Hello, I've been away from postgres for several years, so please forgive me if I forgot nearly everything:-) I've just inherited a database collecting environmental data. There's a background process continually inserting records (not so often, to say the truth) and a web interface to query

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Kevin Grittner
Nico Sabbi nicola.sa...@poste.it wrote: Can you give any hint on the configuration and on the underlying hardware? Well, this particular web site has millions of hits per day (running up to about 20 queries per hit) from thousands of concurrent web users, while accepting logical replication

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Mathis, Jason
I can't speak to the numbers postgresql can or cannot do but the numbers above sound very very doable. If you can get a hold of *greg smith's postgresql high performance*, I always liked his method of tuning buffers and checkpoints using the background writer stats. All of which can help with the

Re: [PERFORM] Configuration tips for very large database

2015-02-12 Thread Claudio Freire
On Thu, Feb 12, 2015 at 7:38 PM, Kevin Grittner kgri...@ymail.com wrote: Nico Sabbi nicola.sa...@poste.it wrote: Queries get executed very very slowly, say 20 minutes. I'd like to know if someone has already succeeded in running postgres with 200-300M records with queries running much faster