I have written a management script to migrate data from an existing
site into our new Django-powered site. The script runs fine with one
exception, memory consumption. A few of the tables I need to migrate
have around 250,000 records, and it seems that the Django ORM caches
each record in memory as it migrates each record.
I have been able to work around it by only migrating about 50,000
records at a time. By the time it reaches 50,000 records the python
process has consumed about 2 GB of RAM. If I let the process end,
python releases all that RAM, then I start a new management command
with the next set of 50,000.
It seems there's probably a better way to manage this memory problem
but I'm a bit of a Python/Django newb. Any insight would be greatly
appreciated. I'm running the migration script on my MacBookPro with
2GB of RAM and 30GB free HD at the moment.
Here's an example of how I've set up my migration script. I cycle over
the ids cause when I tried to do "for row in Legacy.objects.all():" it
tried to swap out like 8GB of RAM then died.
ids = [x[0] for x in Legacy.objects.order_by('time').values_list('id')
[0:50000]]
for row_id in ids:
row = Legacy.objects.get(id=row_id)
new = ShinyNewObj()
new.stuff = row.oldstuff
new.save()
And here's the error I get when it runs out of memory:
-----
--> Importing #172000 id:172000 date:'2008-11-06 10:19:49'
Python(4125) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
File "manage.py", line 27, in <module>
execute_from_command_line()
File "/Library/Python/2.5/site-packages/django/core/management/
__init__.py", line 331, in execute_from_command_line
utility.execute()
File "/Library/Python/2.5/site-packages/django/core/management/
__init__.py", line 295, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Library/Python/2.5/site-packages/django/core/management/
base.py", line 77, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Library/Python/2.5/site-packages/django/core/management/
base.py", line 96, in execute
output = self.handle(*args, **options)
File "/Library/Python/2.5/site-packages/django/core/management/
base.py", line 178, in handle
return self.handle_noargs(**options)
File "/Users/derek/projects/my_project/apps/legacy/management/
commands/migrate_legacy.py", line 161, in handle_noargs
obj._save()
File "/Users/derek/projects/my_project/apps/ShinyNew/models.py",
line 286, in _save
super(ShinyNew, self).save(**kwargs)
File "/Library/Python/2.5/site-packages/django/db/models/base.py",
line 307, in save
self.save_base(force_insert=force_insert,
force_update=force_update)
File "/Library/Python/2.5/site-packages/django/db/models/base.py",
line 354, in save_base
manager.filter(pk=pk_val).extra(select={'a': 1}).values
('a').order_by())):
File "/Library/Python/2.5/site-packages/django/db/models/query.py",
line 185, in __nonzero__
iter(self).next()
File "/Library/Python/2.5/site-packages/django/db/models/query.py",
line 179, in _result_iter
self._fill_cache()
File "/Library/Python/2.5/site-packages/django/db/models/query.py",
line 612, in _fill_cache
self._result_cache.append(self._iter.next())
File "/Library/Python/2.5/site-packages/django/db/models/query.py",
line 653, in iterator
for row in self.query.results_iter():
File "/Library/Python/2.5/site-packages/django/db/models/sql/
query.py", line 206, in results_iter
for rows in self.execute_sql(MULTI):
File "/Library/Python/2.5/site-packages/django/db/models/sql/
query.py", line 1700, in execute_sql
cursor.execute(sql, params)
File "/Library/Python/2.5/site-packages/django/db/backends/util.py",
line 19, in execute
return self.cursor.execute(sql, params)
File "/Library/Python/2.5/site-packages/django/db/backends/mysql/
base.py", line 83, in execute
return self.cursor.execute(query, args)
File "build/bdist.macosx-10.5-i386/egg/MySQLdb/cursors.py", line
166, in execute
File "build/bdist.macosx-10.5-i386/egg/MySQLdb/connections.py", line
35, in defaulterrorhandler
_mysql_exceptions.InterfaceError: (0, '')
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
