[PERFORM] Why is my stats collector so busy?

2009-06-05 Thread Laszlo Nagy
On a 8 processor system, my stats collector is always at 100% CPU. Meanwhile disk I/O is very low. We have many databases, they are accessed frequently. Sometimes there are big table updates, but in most of the time only simple queries are ran against the databases, returning a few records

Re: [PERFORM] Why is my stats collector so busy?

2009-06-05 Thread Bruce Momjian
Laszlo Nagy wrote: On a 8 processor system, my stats collector is always at 100% CPU. Meanwhile disk I/O is very low. We have many databases, they are accessed frequently. Sometimes there are big table updates, but in most of the time only simple queries are ran against the databases,

Re: [PERFORM] Why is my stats collector so busy?

2009-06-05 Thread Tom Lane
Laszlo Nagy gand...@shopzeus.com writes: On a 8 processor system, my stats collector is always at 100% CPU. What platform? What Postgres version? regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your

Re: [PERFORM] Why is my stats collector so busy?

2009-06-05 Thread Merlin Moncure
On Fri, Jun 5, 2009 at 9:38 AM, Bruce Momjianbr...@momjian.us wrote: Laszlo Nagy wrote: On a 8 processor system, my stats collector is always at 100% CPU. Meanwhile disk I/O is very low. We have many databases, they are accessed frequently. Sometimes there are big table updates, but in most

Re: [PERFORM] Scalability in postgres

2009-06-05 Thread Kevin Grittner
Mark Mielke m...@mark.mielke.cc wrote: Kevin Grittner wrote: James Mansion ja...@mansionfamily.plus.com wrote: Kevin Grittner wrote: Sure, but the architecture of those products is based around all the work being done by engines which try to establish affinity to different CPUs, and loop

Re: [PERFORM] Scalability in postgres

2009-06-05 Thread Kevin Grittner
Scott Carey sc...@richrelevance.com wrote: If you wake up 10,000 threads, and they all can get significant work done before yielding no matter what order they run, the system will scale extremely well. But with roughly twice the average response time you would get throttling active requests

Re: [PERFORM] Scalability in postgres

2009-06-05 Thread Robert Haas
On Fri, Jun 5, 2009 at 12:33 AM, da...@lang.hm wrote: On Fri, 5 Jun 2009, Greg Smith wrote: On Thu, 4 Jun 2009, Robert Haas wrote: That's because this thread has altogether too much theory and altogether too little gprof. But running benchmarks and profiling is actual work; that's so much

Re: [PERFORM] Scalability in postgres

2009-06-05 Thread Craig James
Greg Smith wrote: No amount of theoretical discussion advances that any until you're at least staring at a very specific locking problem you've already characterized extensively via profiling. And even then, profiling trumps theory every time. In theory, there is no difference between

Re: [PERFORM] Scalability in postgres

2009-06-05 Thread Greg Smith
On Fri, 5 Jun 2009, Mark Mielke wrote: I disagree that profiling trumps theory every time. That's an interesting theory. Unfortunately, profiling shows it doesn't work that way. Let's see if I can summarize the state of things a bit better here: 1) PostgreSQL stops working as efficiently

Re: [PERFORM] Scalability in postgres

2009-06-05 Thread Greg Smith
On Thu, 4 Jun 2009, Mark Mielke wrote: At it's very simplest, this is the difference between wake one thread (which is then responsible for waking the next thread) vs wake all threadsAny system which actually wakes all threads will probably exhibit scaling limitations. The prototype

Re: [PERFORM] Why is my stats collector so busy?

2009-06-05 Thread Robert Schnabel
On Fri, Jun 5, 2009 at 9:38 AM, Bruce Momjianbr...@momjian.us wrote: Laszlo Nagy wrote: On a 8 processor system, my stats collector is always at 100% CPU. Meanwhile disk I/O is very low. We have many databases, they are accessed frequently. Sometimes there are big table updates, but

Re: [PERFORM] GiST index performance

2009-06-05 Thread Robert Haas
On Thu, Jun 4, 2009 at 12:33 PM, Matthew Wakelingmatt...@flymine.org wrote: Do you have a recommendation for how to go about profiling Postgres, what profiler to use, etc? I'm running on Debian Linux x86_64. I mostly compile with --enable-profiling and use gprof. I know Tom Lane has had

Re: [PERFORM] Scalability in postgres

2009-06-05 Thread Robert Haas
On Fri, Jun 5, 2009 at 1:02 PM, Greg Smithgsm...@gregsmith.com wrote: On Fri, 5 Jun 2009, Mark Mielke wrote: I disagree that profiling trumps theory every time. That's an interesting theory.  Unfortunately, profiling shows it doesn't work that way. I had a laugh when I read this, but I can

Re: [PERFORM] Bad Plan for Questionnaire-Type Query

2009-06-05 Thread David Blewett
On Mon, May 25, 2009 at 11:22 AM, David Blewett da...@dawninglight.net wrote: On Sun, May 24, 2009 at 2:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: It still feels like this schema design is obscuring correlations that the planner needs to know about in order to make decent estimates. I'm not

Re: [PERFORM] Bad Plan for Questionnaire-Type Query

2009-06-05 Thread Josh Berkus
David, My first thought would be to increase statistics dramatically on the filtered columns in hopes of making PG realize there's a lot of rows there; it's off by 8x. Correlations stats are an ongoing issue in PostgreSQL. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com -- Sent

Re: [PERFORM] Bad Plan for Questionnaire-Type Query

2009-06-05 Thread David Blewett
On Fri, Jun 5, 2009 at 7:32 PM, Josh Berkus j...@agliodbs.com wrote: My first thought would be to increase statistics dramatically on the filtered columns in hopes of making PG realize there's a lot of rows there; it's off by 8x.  Correlations stats are an ongoing issue in PostgreSQL. I

Re: [PERFORM] degenerate performance on one server of 3

2009-06-05 Thread Erik Aronesty
See ALTER TABLE and CREATE TABLE  (and the Index variants). ALTER TABLE foo SET (fillfactor=90); I'll try that. This will leave on average, 10% of every 8k block empty and allow updates to columns to more likely live within the same block. Good for the items table. Probably bad for the

Re: [PERFORM] Bad Plan for Questionnaire-Type Query

2009-06-05 Thread Robert Haas
On Fri, Jun 5, 2009 at 8:29 PM, David Blewettda...@dawninglight.net wrote: On Fri, Jun 5, 2009 at 7:32 PM, Josh Berkus j...@agliodbs.com wrote: My first thought would be to increase statistics dramatically on the filtered columns in hopes of making PG realize there's a lot of rows there; it's

Re: [PERFORM] degenerate performance on one server of 3

2009-06-05 Thread Robert Haas
On Thu, Jun 4, 2009 at 7:31 AM, Erik Aronestye...@q32.com wrote: is there some way to view the level of bloat that needs full in each table, so i could write a script that alerts me to the need of a vacuum full  without waiting for random queries to get slow? looking at the results of the