Try doing Legacy.objects.all().iterator(), this isn't technically the
public API, but this grabs the results at a lower level, so it doesn't
get cached.
On Nov 23, 5:44 pm, derek73 <[EMAIL PROTECTED]> wrote:
> I have written a management script to migrate data from an existing
> site into our new Django-powered site. The script runs fine with one
> exception, memory consumption. A few of the tables I need to migrate
> have around 250,000 records, and it seems that the Django ORM caches
> each record in memory as it migrates each record.
>
> I have been able to work around it by only migrating about 50,000
> records at a time. By the time it reaches 50,000 records the python
> process has consumed about 2 GB of RAM. If I let the process end,
> python releases all that RAM, then I start a new management command
> with the next set of 50,000.
>
> It seems there's probably a better way to manage this memory problem
> but I'm a bit of a Python/Django newb. Any insight would be greatly
> appreciated. I'm running the migration script on my MacBookPro with
> 2GB of RAM and 30GB free HD at the moment.
>
> Here's an example of how I've set up my migration script. I cycle over
> the ids cause when I tried to do "for row in Legacy.objects.all():" it
> tried to swap out like 8GB of RAM then died.
>
> ids = [x[0] for x in Legacy.objects.order_by('time').values_list('id')
> [0:50000]]
> for row_id in ids:
> row = Legacy.objects.get(id=row_id)
> new = ShinyNewObj()
> new.stuff = row.oldstuff
> new.save()
>
> And here's the error I get when it runs out of memory:
>
> -----
>
> --> Importing #172000 id:172000 date:'2008-11-06 10:19:49'
> Python(4125) malloc: *** mmap(size=16777216) failed (error code=12)
> *** error: can't allocate region
> File "manage.py", line 27, in <module>
> execute_from_command_line()
> File "/Library/Python/2.5/site-packages/django/core/management/
> __init__.py", line 331, in execute_from_command_line
> utility.execute()
> File "/Library/Python/2.5/site-packages/django/core/management/
> __init__.py", line 295, in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
> File "/Library/Python/2.5/site-packages/django/core/management/
> base.py", line 77, in run_from_argv
> self.execute(*args, **options.__dict__)
> File "/Library/Python/2.5/site-packages/django/core/management/
> base.py", line 96, in execute
> output = self.handle(*args, **options)
> File "/Library/Python/2.5/site-packages/django/core/management/
> base.py", line 178, in handle
> return self.handle_noargs(**options)
> File "/Users/derek/projects/my_project/apps/legacy/management/
> commands/migrate_legacy.py", line 161, in handle_noargs
> obj._save()
> File "/Users/derek/projects/my_project/apps/ShinyNew/models.py",
> line 286, in _save
> super(ShinyNew, self).save(**kwargs)
> File "/Library/Python/2.5/site-packages/django/db/models/base.py",
> line 307, in save
> self.save_base(force_insert=force_insert,
> force_update=force_update)
> File "/Library/Python/2.5/site-packages/django/db/models/base.py",
> line 354, in save_base
> manager.filter(pk=pk_val).extra(select={'a': 1}).values
> ('a').order_by())):
> File "/Library/Python/2.5/site-packages/django/db/models/query.py",
> line 185, in __nonzero__
> iter(self).next()
> File "/Library/Python/2.5/site-packages/django/db/models/query.py",
> line 179, in _result_iter
> self._fill_cache()
> File "/Library/Python/2.5/site-packages/django/db/models/query.py",
> line 612, in _fill_cache
> self._result_cache.append(self._iter.next())
> File "/Library/Python/2.5/site-packages/django/db/models/query.py",
> line 653, in iterator
> for row in self.query.results_iter():
> File "/Library/Python/2.5/site-packages/django/db/models/sql/
> query.py", line 206, in results_iter
> for rows in self.execute_sql(MULTI):
> File "/Library/Python/2.5/site-packages/django/db/models/sql/
> query.py", line 1700, in execute_sql
> cursor.execute(sql, params)
> File "/Library/Python/2.5/site-packages/django/db/backends/util.py",
> line 19, in execute
> return self.cursor.execute(sql, params)
> File "/Library/Python/2.5/site-packages/django/db/backends/mysql/
> base.py", line 83, in execute
> return self.cursor.execute(query, args)
> File "build/bdist.macosx-10.5-i386/egg/MySQLdb/cursors.py", line
> 166, in execute
> File "build/bdist.macosx-10.5-i386/egg/MySQLdb/connections.py", line
> 35, in defaulterrorhandler
> _mysql_exceptions.InterfaceError: (0, '')
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
