os: win32
db: oracle 10
django: 1.3
python: 2.7
While inserting around 750K of records I get the following error
trades done 661000 in 4007.30800009
trades done 662000 in 4011.90100002
Traceback (most recent call last):
File "manage.py", line 14, in <module>
execute_manager(settings)
File "c:\python27\lib\site-packages\django\core\management
\__init__.py", line 438, in execute_manager
utility.execute()
File "c:\python27\lib\site-packages\django\core\management
\__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "c:\python27\lib\site-packages\django\core\management\base.py",
line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "c:\python27\lib\site-packages\django\core\management\base.py",
line 220, in execute
output = self.handle(*args, **options)
File "F:\work\control\src\control\migration\management\commands
\convert.py", line 33, in handle
print conversion.execute()
File "F:\work\control\src\control\..\control\migration\models.py",
line 341, in execute
return PROCESSOR(self, **kwa)
File "F:\work\control\src\control\staging\models.py", line 96, in
extract
trade.save()
File "c:\python27\lib\site-packages\django\db\models\base.py", line
460, in save
self.save_base(using=using, force_insert=force_insert,
force_update=force_update)
File "c:\python27\lib\site-packages\django\db\models\base.py", line
553, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
File "c:\python27\lib\site-packages\django\db\models\manager.py",
line 195, in _insert
return insert_query(self.model, values, **kwargs)
File "c:\python27\lib\site-packages\django\db\models\query.py", line
1436, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "c:\python27\lib\site-packages\django\db\models\sql
\compiler.py", line 791, in execute_sql
cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "c:\python27\lib\site-packages\django\db\models\sql
\compiler.py", line 735, in execute_sql
cursor.execute(sql, params)
File "c:\python27\lib\site-packages\django\db\backends\util.py",
line 43, in execute
logger.debug('(%.3f) %s; args=%s' % (duration, sql, params),
MemoryError
In my code I've been careful to only use iterators, etc and should
only be using enough memory for one record (around 1k). While doing
inserts is there any accumulation of objects going on ? For instance
is commit only occurring after n saves.
here is a stripped down version.
@classmethod
def extract(cls, conversion, migration_run_id = 6711):
"""
This code is used to migrate trades from one trading system to
another.
"""
tick = time.time()
run = Job.objects.get(name=cls.JOB_NAME).start()
if migration_run_id is None:
migration_run_id = run.pk
_, src_cursor = conversion.source_sql.execute() # getting a
cursor on the source database (using sqlachemy against mssql here )
count = 0
for record in src_cursor:
count +=1
fields = conversion.source_sql.source.to_dict(src_cursor,
record) # justs converts a record to a dict.
fields['run'] = run
fields['migration_run'] = migration_run_id
trade = cls(**fields) # creating a django model
trade.save() # here we run out of memory after about
650,000 inserts. All records where committed before crash.
if count % 1000 == 0:
print 'trades done', count, 'in ', time.time() - tick
print 'trades done', count, 'in ', time.time() - tick
return migration_run_id, run.pk, count
Thanks
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
