Some years ago I got a similar problem using postgresql. To.improve the bulk performance I chose to disable all keys and indexes after bulk the data. On postgres it solved my problem, but if you need more performance, and have more than one computer, I recommend that you take a look at Cassandra, it has a grate insert performance and could be used to hold that large data.
Take a look at this post, I wrote that in Portuguese, but google can translate it: Http://imarcelolz.blogspot.com/2011/11/cassandra-hadoopy-performance-tunning.html Ivo Marcelo Leonardi Zaniolo +55 71 9302 3400 imarcel...@gmail.com www.informatizzare.com.br imarcelolz.blogspot.com.br Em 08/08/2012 01:18, "Cal Leeming [Simplicity Media Ltd]" < cal.leem...@simplicitymedialtd.co.uk> escreveu: > Hi Anton, > > In short, attempting to do any sort of bulk import "out of the box" with > the ORM, will always end with bad performance. > > No matter how fast your disks are (SSDs with 4000 iops in RAID 1 for > example), you'll still only get around 0.1s per insert via the ORM on a > single thread, and if you multi thread the import on a MySQL backend then > you'll end up hitting the notorious high throughput race condition ( > https://code.djangoproject.com/ticket/18557 ) > > The method used to speed up your bulk imports depend entirely on the > actual data itself. > > Depending on whether or not you absolutely need all the functionality of > the ORM during import (signals, hooks etc) - then you could also look at > using bulk_create() that comes with Django 1.4. > > I actually did a webcast about a year ago on how to do bulk updates to > achieve massively increased throughput (from 30 rows/sec to 8000 rows/sec) > - and this method was eventually integrated into the DSE plugin ( > http://pypi.python.org/pypi/dse/3.3.0 - look for bulk_update ). > > As your table and indexes grow, both read/write performance will slowly > start to get worse once you pass the 1mil rows point - so make sure to put > some thought into the initial model structure (using ints to define choices > where possible, no indexes on varchar/char if possible, make good use of > composite/compound indexes etc). > > Failing that, if you want to drag every single microsecond of performance > out of this, then you could performance the bulk imports as raw SQL. > > Hope this helps! > > Cal > > On Mon, Aug 6, 2012 at 4:14 PM, creecode <creec...@gmail.com> wrote: > >> Hello Anton, >> >> On Friday, August 3, 2012 9:05:47 AM UTC-7, Anton wrote: >> >> >>> My Problem: I will populate it with 50000 items. >>> >>> Has Django really such a bad performance for data insertion? >>> >>> I can't believe it, so ... can somebody give me a hint? >>> Is there a doc page dedicated to speed/performance issues?? >>> >>> Otherwise I have to look for an other system. >>> >> >> Before you go looking for another system you may want to do some research >> on inserting lots of rows in Django. Cal Leeming on this list has had >> some experience dealing with lots of rows and there are several threads on >> the topic. Check the Google interface for this group and use the search >> feature. Also try Googling for inserting lots of data with Django, several >> folks have written about their experiences inserting lots of rows. >> >> Toodle-loooooooooooooo................... >> creecode >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/django-users/-/yLxStReK_1gJ. >> >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to >> django-users+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-users?hl=en. >> > > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
