Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: Tom Lane t...@sss.pgh.pa.us wrote: Any sane text search application is going to try to filter out common words as stopwords; it's only the failure to do that that's making this run slow. I'd rather have

Re: [PERFORM] Problem with database performance, Debian 4gb ram ?

2009-11-03 Thread Kevin Grittner
Grant Masan grant.mas...@gmail.com wrote: max_connections = 80 shared_buffers = 512MB temp_buffers = 8MB work_mem = 20MB maintenance_work_mem = 384MB wal_buffers = 8MB checkpoint_segments = 128MB effective_cache_size = 2304MB checkpoint_timeout = 1h Pending further information, these

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: Perhaps I'm missing something. My point was that there are words which are too common to be useful for index searches, yet uncommon enough to usefully limit the results. These words could typically benefit from tsearch2 style parsing and

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: The answer to that clearly is to not index common terms My understanding is that we don't currently get statistics on how common the terms in a tsvector column are until we ANALYZE the *index* created from it. Seems like sort of a Catch 22. Also, if we

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: I'm wondering if anyone has ever confirmed that probing for the more frequent term through the index is *ever* a win, versus using the index for the most common of the top level AND conditions and doing the rest on recheck. s/most/least/

Re: [PERFORM] Problem with database performance, Debian 4gb ram ?

2009-11-03 Thread Jeff Janes
On Tue, Nov 3, 2009 at 7:13 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Grant Masan grant.mas...@gmail.com wrote: cpu_tuple_cost = 0.0030 cpu_index_tuple_cost = 0.0010 cpu_operator_cost = 0.0005 Why did you make these adjustments? I usually have to change the ratio between page

[PERFORM] Free memory usage Sol10, 8.2.9

2009-11-03 Thread Subbiah Stalin-XCGF84
All, I'm trying to understand the free memory usage and why it falls below 17G sometimes and what could be causing it. Any pointers would be appreciated. r...@prod1 # prtconf System Configuration: Sun Microsystems sun4u Memory size: 32768 Megabytes [postg...@prod1 ~]$ vmstat 5 10 kthr

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Jesper Krogh
Tom Lane wrote: It may well be that Jesper's identified a place where the GIN code could be improved --- it seems like having the top-level search logic be more aware of the AND/OR structure of queries would be useful. But the particular example shown here doesn't make a very good case for

Re: [PERFORM] Optimizer + bind variables

2009-11-03 Thread Craig Ringer
David Kerr wrote: Does/is it possible for the PG optimizer come up with differnet plans when you're using bind variables vs when you send static values? Yes, if the bind variable form causes your DB access driver to use a server-side prepared statement. Pg can't use its statistics to improve

[PERFORM] Optimizer + bind variables

2009-11-03 Thread David Kerr
Does/is it possible for the PG optimizer come up with differnet plans when you're using bind variables vs when you send static values? like if my query was select * from users (add a bunch of complex joins) where username = 'dave' vs select * from users (add a bunch of complex joins) where

Re: [PERFORM] Free memory usage Sol10, 8.2.9

2009-11-03 Thread Jeremy Harris
On 11/03/2009 07:16 PM, Subbiah Stalin-XCGF84 wrote: All, I'm trying to understand the free memory usage and why it falls below 17G sometimes and what could be causing it. Any pointers would be appreciated. r...@prod1 # prtconf System Configuration: Sun Microsystems sun4u Memory size: 32768

Re: [PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Craig Ringer
Brian Karlak wrote: The setup is relatively simple: there is a central queue table in postgres. Worker daemons do a bounded, ordered, limited SELECT to grab a row, which they lock by setting a value in the queue.status column. You can probably do an UPDATE ... RETURNING to turn that into

[PERFORM] High Frequency Inserts to Postgres Database vs Writing to a File

2009-11-03 Thread Jay Manni
Hi: I have an application wherein a process needs to read data from a stream and store the records for further analysis and reporting. The data in the stream is in the form of variable length records with clearly defined fields - so it can be stored in a database or in a file. The only caveat

Re: [PERFORM] Optimizer + bind variables

2009-11-03 Thread David Kerr
On Wed, Nov 04, 2009 at 07:43:16AM +0800, Craig Ringer wrote: - David Kerr wrote: - Does/is it possible for the PG optimizer come up with differnet plans when - you're using bind variables vs when you send static values? - - Yes, if the bind variable form causes your DB access driver to use a

Re: [PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Brian Karlak
On Nov 3, 2009, at 4:03 PM, Craig Ringer wrote: I don't have a good answer for you there. Perhaps using Pg's locking to do your queueing, rather than updating a status flag, might let you use a cursor? Have a look at the list archives - there's been a fair bit of discussion of queuing

[PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Brian Karlak
Hello All -- I have a simple queuing application written on top of postgres which I'm trying to squeeze some more performance out of. The setup is relatively simple: there is a central queue table in postgres. Worker daemons do a bounded, ordered, limited SELECT to grab a row, which

Re: [PERFORM] Optimizer + bind variables

2009-11-03 Thread David Kerr
On Wed, Nov 04, 2009 at 11:02:22AM +1100, Chris wrote: - David Kerr wrote: - On Wed, Nov 04, 2009 at 07:43:16AM +0800, Craig Ringer wrote: - - David Kerr wrote: - No. - - This is explained in the notes here: - - http://www.postgresql.org/docs/current/static/sql-prepare.html sigh and i've read

Re: [PERFORM] High Frequency Inserts to Postgres Database vs Writing to a File

2009-11-03 Thread Scott Marlowe
On Tue, Nov 3, 2009 at 8:12 PM, Jay Manni jma...@fireeye.com wrote: Hi: I have an application wherein a process needs to read data from a stream and store the records for further analysis and reporting. The data in the stream is in the form of variable length records with clearly defined

Re: [PERFORM] High Frequency Inserts to Postgres Database vs Writing to a File

2009-11-03 Thread David Saracini
could be several 1000 records a second. So, are there periods when there are no/few records coming in? Do the records/data/files really need to be persisted? The following statement makes me think you should go the flat file route: The advantage of running complex queries to mine the data

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
I wrote: Tom Lane t...@sss.pgh.pa.us wrote: But the particular example shown here doesn't make a very good case for that, because it's hard to tell how much of a penalty would be taken in more realistic examples. Fair enough. We're in the early stages of moving to tsearch2 and I

Re: [PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Tom Lane
Brian Karlak zen...@metaweb.com writes: My question is this: is there some way that I can keep a cursor / pointer / reference / whatever to the row I fetched originally, so that I don't have to search for it again when I'm ready to write results? If you don't expect any updates to the

Re: [PERFORM] Problem with database performance, Debian 4gb ram ?

2009-11-03 Thread Kevin Grittner
Jeff Janes jeff.ja...@gmail.com wrote: On Tue, Nov 3, 2009 at 7:13 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Grant Masan grant.mas...@gmail.com wrote: cpu_tuple_cost = 0.0030 cpu_index_tuple_cost = 0.0010 cpu_operator_cost = 0.0005 Why did you make these adjustments? I