Re: [PERFORM] ETL optimization

2005-06-27 Thread Bricklen Anderson
Dennis Bjorklund wrote: > On Thu, 23 Jun 2005, Bricklen Anderson wrote: > > >>iii. UNIQUE constraint on table "t1". This didn't seem to perform too >>badly with fewer rows (preliminary tests), but as you'd expect, on error >>the whole transaction would roll back. Is it possible to skip a row if >

Re: [PERFORM] ETL optimization

2005-06-23 Thread Dennis Bjorklund
On Thu, 23 Jun 2005, Bricklen Anderson wrote: > iii. UNIQUE constraint on table "t1". This didn't seem to perform too > badly with fewer rows (preliminary tests), but as you'd expect, on error > the whole transaction would roll back. Is it possible to skip a row if > it causes an error, as opposed

Re: [PERFORM] ETL optimization

2005-06-23 Thread Bricklen Anderson
Jacques Caron wrote: > > I have a similar situation, and the solution I use (though I haven't > really tested many different situations): > - have a trigger ON INSERT which does: > UPDATE set whatever_value=NEW.whatever_value,... WHERE > whatever_key=NEW.whatever.key AND... > IF FOUND THEN > RETU

Re: [PERFORM] ETL optimization

2005-06-23 Thread Jacques Caron
Hi, At 21:38 23/06/2005, Bricklen Anderson wrote: Situation: I'm trying to optimize an ETL process with many upserts (~100k aggregated rows) (no duplicates allowed). The source (table t2) table holds around 14 million rows, and I'm grabbing them 100,000 rows at a time from t2, resulting in abo

Re: [PERFORM] ETL optimization

2005-06-23 Thread Bricklen Anderson
Meetesh Karia wrote: > I don't know what this will change wrt how often you need to run VACUUM > (I'm a SQL Server guy), but instead of an update and insert, try a > delete and insert. You'll only have to find the duplicate rows once and > your insert doesn't need a where clause. > > Meetesh > V

Re: [PERFORM] ETL optimization

2005-06-23 Thread Meetesh Karia
I don't know what this will change wrt how often you need to run VACUUM (I'm a SQL Server guy), but instead of an update and insert, try a delete and insert.  You'll only have to find the duplicate rows once and your insert doesn't need a where clause. MeeteshOn 6/23/05, Bricklen Anderson <[EMAIL

[PERFORM] ETL optimization

2005-06-23 Thread Bricklen Anderson
Situation: I'm trying to optimize an ETL process with many upserts (~100k aggregated rows) (no duplicates allowed). The source (table t2) table holds around 14 million rows, and I'm grabbing them 100,000 rows at a time from t2, resulting in about 100,000 distinct rows in the destination table (t1).