Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-27 Thread Scott Marlowe
On Mon, Nov 26, 2012 at 12:46 AM, Heikki Linnakangas wrote: > On 25.11.2012 18:30, Catalin Iacob wrote: >> >> So it seems we're just doing too many connections and too many >> queries. Each page view from a user translates to multiple requests to >> the application server and each of those transla

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-25 Thread Heikki Linnakangas
On 25.11.2012 18:30, Catalin Iacob wrote: So it seems we're just doing too many connections and too many queries. Each page view from a user translates to multiple requests to the application server and each of those translates to a connection and at least a few queries (which are done in middlew

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-25 Thread Catalin Iacob
Thanks to everybody for their help, sorry for not getting back earlier but available time shrunk very quickly as the deadline approached and afterwards this kind of slipped off my mind. On Tue, Nov 6, 2012 at 12:31 AM, Jeff Janes wrote: > It still has something to contribute if connections are ma

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-05 Thread Jeff Janes
On Mon, Nov 5, 2012 at 3:58 PM, Marko Kreen wrote: > On Tue, Nov 6, 2012 at 1:31 AM, Jeff Janes wrote: >> On Mon, Nov 5, 2012 at 2:58 PM, Marko Kreen wrote: >>> On Sun, Nov 4, 2012 at 1:53 AM, Jeff Janes wrote: On a 4 CPU machine, if I run pgbench -c10 -j10 with dummy queries (like "s

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-05 Thread Marko Kreen
On Tue, Nov 6, 2012 at 1:31 AM, Jeff Janes wrote: > On Mon, Nov 5, 2012 at 2:58 PM, Marko Kreen wrote: >> On Sun, Nov 4, 2012 at 1:53 AM, Jeff Janes wrote: >>> On a 4 CPU machine, if I run pgbench -c10 -j10 with dummy queries >>> (like "select 1;" or "set timezone...") against 2 instances of >>>

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-05 Thread Jeff Janes
On Mon, Nov 5, 2012 at 2:58 PM, Marko Kreen wrote: > On Sun, Nov 4, 2012 at 1:53 AM, Jeff Janes wrote: >> On a 4 CPU machine, if I run pgbench -c10 -j10 with dummy queries >> (like "select 1;" or "set timezone...") against 2 instances of >> pgbouncer, I get nearly twice the throughput as if I use

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-05 Thread Marko Kreen
On Sun, Nov 4, 2012 at 1:53 AM, Jeff Janes wrote: > On a 4 CPU machine, if I run pgbench -c10 -j10 with dummy queries > (like "select 1;" or "set timezone...") against 2 instances of > pgbouncer, I get nearly twice the throughput as if I use only one > instance. > > A rather odd workload, maybe, b

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-05 Thread Merlin Moncure
On Sat, Nov 3, 2012 at 6:53 PM, Jeff Janes wrote: > On Wed, Oct 31, 2012 at 11:39 AM, Merlin Moncure wrote: >> On Tue, Oct 30, 2012 at 4:58 PM, Jeff Janes wrote: >>> On Mon, Oct 29, 2012 at 5:11 PM, Catalin Iacob >>> wrote: >>> pgbouncer 1.4.2 installed from Ubuntu's packages on the same

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-11-03 Thread Jeff Janes
On Wed, Oct 31, 2012 at 11:39 AM, Merlin Moncure wrote: > On Tue, Oct 30, 2012 at 4:58 PM, Jeff Janes wrote: >> On Mon, Oct 29, 2012 at 5:11 PM, Catalin Iacob >> wrote: >> >>> pgbouncer 1.4.2 installed from Ubuntu's packages on the same machine >>> as Postgres. Django connects via TCP/IP to pgb

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-31 Thread Merlin Moncure
On Tue, Oct 30, 2012 at 4:58 PM, Jeff Janes wrote: > On Mon, Oct 29, 2012 at 5:11 PM, Catalin Iacob wrote: > >> pgbouncer 1.4.2 installed from Ubuntu's packages on the same machine >> as Postgres. Django connects via TCP/IP to pgbouncer (it does one >> connection and one transaction per request)

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-31 Thread Jeff Janes
On Tue, Oct 30, 2012 at 3:16 PM, Scott Marlowe wrote: > On Tue, Oct 30, 2012 at 4:11 PM, Greg Williamson > wrote: >> Jeff / Catalin -- >> >> Jeff Janes wrote: >> >>>On Mon, Oct 29, 2012 at 5:11 PM, Catalin Iacob >>>wrote: >>> pgbouncer 1.4.2 installed from Ubuntu's packages on the same mac

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-30 Thread Scott Marlowe
On Tue, Oct 30, 2012 at 4:11 PM, Greg Williamson wrote: > Jeff / Catalin -- > > Jeff Janes wrote: > >>On Mon, Oct 29, 2012 at 5:11 PM, Catalin Iacob wrote: >> >>> pgbouncer 1.4.2 installed from Ubuntu's packages on the same machine >>> as Postgres. Django connects via TCP/IP to pgbouncer (it does

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-30 Thread Greg Williamson
Jeff / Catalin -- Jeff Janes wrote: >On Mon, Oct 29, 2012 at 5:11 PM, Catalin Iacob wrote: > >> pgbouncer 1.4.2 installed from Ubuntu's packages on the same machine >> as Postgres. Django connects via TCP/IP to pgbouncer (it does one >> connection and one transaction per request) and pgbouncer k

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-30 Thread Jeff Janes
On Mon, Oct 29, 2012 at 5:11 PM, Catalin Iacob wrote: > pgbouncer 1.4.2 installed from Ubuntu's packages on the same machine > as Postgres. Django connects via TCP/IP to pgbouncer (it does one > connection and one transaction per request) and pgbouncer keeps > connections open to Postgres via Uni

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-30 Thread Kevin Grittner
Shaun Thomas wrote: > On 10/30/2012 06:55 AM, Kevin Grittner wrote: >> Is there a good transaction-based connection pooler in Python? >> You're better off with a good pool built in to the client >> application than with a good pool running as a separate process >> between the client and the databa

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-30 Thread Shaun Thomas
On 10/30/2012 06:55 AM, Kevin Grittner wrote: Is there a good transaction-based connection pooler in Python? You're better off with a good pool built in to the client application than with a good pool running as a separate process between the client and the database, IMO. Could you explain thi

Re: [PERFORM] How to keep queries low latency as concurrency increases

2012-10-30 Thread Kevin Grittner
Catalin Iacob wrote: > Hardware: > Virtual machine running on top of VMWare > 4 cores, Intel(R) Xeon(R) CPU E5645 @ 2.40GHz > 4GB of RAM You should carefully test transaction-based pools limited to around 8 DB connections. Experiment with different size limits. http://wiki.postgresql.org/wiki/Nu

[PERFORM] How to keep queries low latency as concurrency increases

2012-10-29 Thread Catalin Iacob
As I increase concurrency I'm experiencing what I believe are too slow queries given the minuscule amount of data in my tables. I have 20 Django worker processes and use ab to generate 3000 requests to a particular URL which is doing some read only queries. I ran this with ab concurrency level set