Load the files into a temp table and go from there... COPY ... FROM file; UPDATE existing_table SET ... WHERE ...; INSERT INTO existing_table SELECT * FROM temp_table WHERE NOT EXISTS( SELECT * FROM existing_table WHERE ...)
On Wed, Mar 22, 2006 at 10:32:10AM -0500, ashah wrote: > I have a database with foreign keys enabled on the schema. I receive > different > files, some of them are huge. And I need to load these files in the database > every night. There are several scenerios that I want to design an optimal > solution for - > > 1. One of the file has around 80K records and I have to delete everything > from > the table and load this file. The provider never provides a "delta file" so I > dont have a way to identify which records are already present and which are > new. If I dont delete everything and insert fresh, I have to make around 80K > selects to decide if the records exist or not. Now there are lot of tables > that have foreign keys linked with this table so unless I disable the foreign > keys, I cannot really delete anything from this table. What would be a good > practise here? > > 2. Another file that I receive has around 150K records that I need to load in > the database. Now one of the fields is logically a "foreign key" to another > table, and it is linked to the parent table via a database generated unique > ID instead of the actual value. But the file comes with the actual value. So > once again, I have to either drop the foreign key, or make 150K selects to > determine the serial ID so that the foreign key is satisfied. What would be a > good strategy in this scenerio ? > > Please pardon my inexperience with database ! > > Thanks, > Amit > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org > -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---------------------------(end of broadcast)--------------------------- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
