On Tuesday 13 June 2006 11:44, keukaman wrote: > I'd like to import several thousand records, in CSV format, into an > existing postgres table. Does anyone know a utility that would allow me > to do this?
Hey keukaman, I had to struggle with a system of regularly importing around 1.5 million records into our database. I cooked up something that does the job. Following is a performance summary and then a link to sample code. Take the following numbers with a really big grain of salt. I didn't record all the exact rates, it's been months since I wrote the script, etc. With that caveat in mind... My first stab processed around 2,000 records per minute (records serialized with Django's orm). Second stab, if I recall correctly processed around 6,000 records per minute (drop indexes first, use Django's orm, replace indexes). The final version[1], a little over 33,000 records per minute. Here's a cleansed and generalized version of the script for your entertainment. WARNING: It *deletes records from the table before importing* the input file data. The usual disclaimers apply: It'll need customizing to run (add your field names, etc.), there's lots of room for improvement, works for me, yaddayaddayadda... [1] <http://django.pastebin.com/707965> I hope it helps. Eric. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~----------~----~----~----~------~----~------~--~---
