[PERFORM] query optimization

2009-11-23 Thread Faheem Mitha
Hi everybody, I've got two queries that needs optimizing. Actually, there are others, but these are pretty representative. You can see the queries and the corresponding plans at http://bulldog.duhs.duke.edu/~faheem/snpdb/opt.pdf or http://bulldog.duhs.duke.edu/~faheem/snpdb/opt.tex if

Re: [PERFORM] query optimization

2009-11-23 Thread Faheem Mitha
On Mon, 23 Nov 2009, Thom Brown wrote: Hi Faheem, There appears to be a discrepancy between the 2 PDFs you provided.  One says you're using PostgreSQL 8.3, and the other shows you using common table expressions, which are only available in 8.4+. Yes, sorry. I'm using Postgresql 8.4. I

Re: [PERFORM] query optimization

2009-11-23 Thread Faheem Mitha
, 2009 at 12:52 AM, marcin mank marcin.m...@gmail.com wrote: On Tue, Nov 24, 2009 at 12:49 AM, Faheem Mitha fah...@email.unc.edu wrote: Yes, sorry. I'm using Postgresql 8.4. I guess I should go through diag.pdf and make sure all the information is current. Thanks

Re: [PERFORM] query optimization

2009-11-25 Thread Faheem Mitha
Hi Robert, Thanks very much for your suggestions. On Wed, 25 Nov 2009, Robert Haas wrote: On Mon, Nov 23, 2009 at 5:47 PM, Faheem Mitha fah...@email.unc.edu wrote: Hi everybody, I've got two queries that needs optimizing. Actually, there are others, but these are pretty representative

Re: [PERFORM] query optimization

2009-11-27 Thread Faheem Mitha
On Wed, 25 Nov 2009, Robert Haas wrote: On Wed, Nov 25, 2009 at 5:54 PM, Faheem Mitha fah...@email.unc.edu wrote: Well, any method of DISTINCT-ifying is likely to be somewhat slow, but I've had good luck with SELECT DISTINCT ON (...) in the past, as compared with other methods. YMMV

[PERFORM] experiments in query optimization

2010-03-25 Thread Faheem Mitha
Hi everyone, I've been trying to reduce both memory usage and runtime for a query. Comments/suggestions gratefully received. Details are at http://bulldog.duhs.duke.edu/~faheem/snppy/opt.pdf See particularly Section 1 - Background and Discussion. If you want a text version, see

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Faheem Mitha
On Mon, 29 Mar 2010, Robert Haas wrote: On Thu, Mar 25, 2010 at 3:57 PM, Faheem Mitha fah...@email.unc.edu wrote: Hi everyone, I've been trying to reduce both memory usage and runtime for a query. Comments/suggestions gratefully received. Details are at http://bulldog.duhs.duke.edu

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Faheem Mitha
On Mon, 29 Mar 2010, Robert Haas wrote: On Mon, Mar 29, 2010 at 2:31 PM, Faheem Mitha fah...@email.unc.edu wrote: It's not really too clear to me from reading this what specific questions you're trying to answer. Quote from opt.{tex/pdf}, Section 1: If I have to I can use Section~\ref

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Faheem Mitha
On Tue, 30 Mar 2010, Kevin Grittner wrote: Faheem Mitha fah...@email.unc.edu wrote: If you're concerned about memory usage, try reducing work_mem; you've probably got it set to something huge. work_mem = 1 GB (see diag.{tex/pdf}). The point isn't that I'm using so much memory. Again, my

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Faheem Mitha
)::text || ' '::text) || (dedup_patient_anno.alleleb_id)::text) ELSE NULL::text END), hapmap.geno.idlink_id, hapmap.geno.anno_id, pheno.patientid, pheno.phenotype, sex.code Faheem. On Tue, 30 Mar 2010, Kevin Grittner wrote: Faheem Mitha

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Faheem Mitha
On Wed, 31 Mar 2010, Matthew Wakeling wrote: On Tue, 30 Mar 2010, Faheem Mitha wrote: work_mem = 1 GB (see diag.{tex/pdf}). Sure, but define sane setting, please. I guess part of the point is that I'm trying to keep memory low You're trying to keep memory usage low, but you have

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Wed, 31 Mar 2010, Faheem Mitha wrote: On Tue, 30 Mar 2010, Robert Haas wrote: On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha fah...@email.unc.edu You might need to create some indices, too. Ok. To what purpose? This query picks up everything from the tables and the planner does

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
Hi Eliot, Thanks for the comment. On Thu, 1 Apr 2010, Eliot Gable wrote: On Thu, Apr 1, 2010 at 7:46 AM, Faheem Mitha fah...@email.unc.edu wrote: Looking at this more closely, idlink_id and anno_id are primary keys, so already have indexes on them, so my understanding (from the docs

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Thu, 1 Apr 2010, Robert Haas wrote: On Thu, Apr 1, 2010 at 2:15 PM, Faheem Mitha fah...@email.unc.edu wrote: I had set the foreign keys in question (on the geno table) to be primary keys. This is because this setup is basically a glorified spreadsheet, and I don't want more than one

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Thu, 1 Apr 2010, Eliot Gable wrote: On Thu, Apr 1, 2010 at 3:01 PM, Faheem Mitha fah...@email.unc.edu wrote: So, should I add indexes on the individual foreign key cols idlink_id and anno_id after all? I doubt that would help. You're