Re: [PERFORM] Slow loads when indexes added.

2005-03-16 Thread Stef
[EMAIL PROTECTED] mentioned :
=> Try ANALYZE after loading the referenced tables, but before loading the main 
table

I attached a new script for creating the load file...
 
Analyze didn't help, it actually took longer to load.
I set autocommit to off, and put a commit after every 100
inserts, chattr'd noatime atrribute off recursively on PGDATA, and
set fsync to off, this improved the time from 3min 51sec to 2min 37 sec
for the slow scenario.

But I was already doing all these things  in the app that 
used to take 40 minutes, but now takes four hours to load.

Any other suggestions?

Kind Regards
Stefan

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]


[PERFORM] Slow loads when indexes added.

2005-03-15 Thread Stef
Hi all,

I suspect this problem/bug has been dealt with already, but I couldn't
find anything in the mail archives.

I'm using postgres 7.3, and I managed to recreate the problem using the 
attached 
files. 

The database structure is in slow_structure.sql

After creating the database, using this script, I ran run_before_load__fast.sql

Then I created a load file using create_loadfile.sh (It creates a file called 
load.sql)

I timed the loading of this file, and it loaded in 1 min 11.567 sec

Then I recreated the database from slow_structure.sql, ran 
run_before_load__slow.sql,
and then loaded the same load.sql and it took 3 min 51.293 sec which is about 6 
times slower.

I tried the same thing on postgres 8.0.0 to see if it does the same thing, but 
there it
was consistently slow : 3 min 31.367 sec

The only way I got the load.sql to load fast on postgres 8.0.0,  was by not 
creating
any of the foreign key constraints that point to the "main" table, and then 
enabling them
afterwards. This gave me the fastest time overall  : 1 min 4.911 sec

My problem is that on the postgres 7.3.4 database I'm working with, a load 
process that
used to take 40 minutes, now takes 4 hours, because of 3 rows data being loaded 
into 
a table (similar in setup to the "main" table in the example) before the 
indexes were created.
(This happens automatically  when you dump and re-import the database (7.3.4))

Is there a way to get it to load fast again on the 7.3 database without 
dropping the foreign 
key constraints (After running run_before_load_slow.sql) ?

And, if someone knows off-hand, what's happening here?

TIA
Kind Regards
Stefan#!/bin/bash

for x in one two three four five six seven eight nine ten
do 
   for y in eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen twenty
   do
  for z in `seq 1 100`
  do
echo "insert into main (c_text,d_text) values ('${x}','${y}');" >> load.sql
echo "insert into a(id) values (currval('public.main_id_seq'::text));" >> load.sql
echo "insert into b(id) values (currval('public.main_id_seq'::text));" >> load.sql
echo "insert into e(id) values (currval('public.main_id_seq'::text));" >> load.sql
echo "insert into f(id) values (currval('public.main_id_seq'::text));" >> load.sql
echo "insert into g(id) values (currval('public.main_id_seq'::text));" >> load.sql
echo "insert into h(id) values (currval('public.main_id_seq'::text));" >> load.sql
  done
   done
done


run_before_load__fast.sql
Description: Binary data


run_before_load__slow.sql
Description: Binary data


slow_structure.sql
Description: Binary data

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match