On Fri, Dec 26, 2008 at 2:34 AM, Russell Keith-Magee <[email protected] > wrote:
> > On Thu, Dec 25, 2008 at 9:02 AM, bsergean <[email protected]> wrote: > > > > Hi all, > > > > I'm populating my database manually using a script that create ORM > > objects and save them... I have a lot of datas and it's fairly slow. > > I'm wondering what is the best way to speed this up. > > > > From > http://docs.djangoproject.com/en/dev/howto/initial-data/#howto-initial-data > > I see there are two alternatives. > > > > * Using fixtures > > I could create a big json files (or several), and then load them. Will > > it be smart and do the smallest number of query, or a bunch of > > queries ? > > This will essentially be no different to your 'create ORM objects' > approach. Internally, Django uses ORM objects in the deserialization > process, so if you are seeing problems manually creating ORM objects, > the same problems will probably exist with deserialization. > OK. > > > * Or using raw SQL ? (I'm using postgres, and I saw you can do an > > INSERT command with multiple rows. Is it a good way to speedup > > things) ? > > Assuming you can automate the production of the SQL and, this will > always be the fastest approach. However, the price you pay is the > convenience of dealing with the ORM. You will need to balance the > costs for your own circumstances. > (I'm using postgres) What I'm gonna probably do is generate the same thing that pg_restore want, taking for model the output of pg_dump -F t (tar output is convenient). It uses the COPY statement to bulk load lots of rows from a file, that should be fast. > > A third option that you haven't considered is that you may not be > using Django properly :-) An often forgotten feature of Django is that > When DEBUG is enabled, Django logs every command that goes to the SQL > backend. As a consequence, if you run a long lived script, you will > see a slowdown as memory gets eaten up by the SQL activity log. If you > disable DEBUG during your production data load, logging is disabled, > and you should find that performance improves dramatically. > Good to know, but I had it turned off. (I remember noticing a big performances difference at the app level when I turned DEBUG off). Thanks for your help, - Benjamin. > > Yours > Russ Magee %-) > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
