Re: [PERFORM] Scrub one large table against another

2006-10-12 Thread Brendan Curran
Well, IN != EXISTS != JOIN. Exists just stops as soon as it finds a record. For some cases, it's equivalent to IN, but not all. IN has to de-duplicate it's list in some fashion. For small IN lists, you can do this with an OR, but at some point you need to switch to an actual unique (actually, I

Re: [PERFORM] Scrub one large table against another (vmstat output)

2006-10-11 Thread Brendan Curran
What prevents you from using an aggregate function? I guess I could actually obtain the results in an aggregate function and use those to maintain a summary table. There is a web view that requires 'as accurate as possible' numbers to be queried per group (all 40 groups are displayed on the

Re: [PERFORM] Scrub one large table against another

2006-10-11 Thread Brendan Curran
Tom Lane wrote: Brendan Curran <[EMAIL PROTECTED]> writes: So much time is being spent in the Unique and Sort leaves... I would think that it wouldn't need to do the unique portion, since there is no DISTINCT clause... There's nothing in that query suggesting that suppress

Re: [PERFORM] Scrub one large table against another

2006-10-10 Thread Brendan Curran
Tom Lane wrote: Brendan Curran <[EMAIL PROTECTED]> writes: Tom Lane wrote: Have you tried doing EXPLAIN ANALYZE of each of the INSERT/DELETE steps? FIRST INSERT (Just the select is explained): EXPLAIN ANALYZE, please, not just EXPLAIN. regards, tom lane

Re: [PERFORM] Scrub one large table against another

2006-10-10 Thread Brendan Curran
Tom Lane wrote: Brendan Curran <[EMAIL PROTECTED]> writes: CREATE TEMP TABLE temp_list_suppress(email_record_id int8); INSERT INTO temp_list_suppress SELECT email_record_id from ONLY email_record er WHERE email_list_id = 9 AND email IN (select email from su

[PERFORM] Scrub one large table against another

2006-10-10 Thread Brendan Curran
I currently have a db supporting what is for the most part an OLAP data warehousing application. One table (good data) has roughly 120 million rows, divided into roughly 40 different relational groups (logically by foreign key). Every time I add data to this table, I need to afterwards scrub t