Hi,
I am new web2py user and I have some performance problems with
import_from_csv_file method. First of all i have big collection of data
that i want to upload to Google App Engine. I splited data into 1000 parts,
each contains csv serialized rows - about 1367 rows per file. I am doing
loop to import each file to database using:
def csv_import():
for i in xrange(0, 1000):
file = open(os.path.join(request.folder,'private', 'geonames',
'chunk_' + str(i)), 'r')
db.geonames.import_from_csv_file(file)
db._timings = []
file.close()
As You can see it is rather simple method to achieve this. But the main
problem is that every loop iteration is increasing the overall memory usage
for application. It is never stoped and in 10 iteration it used all system
resources and app is terminated.
I think that with every iteration some objects related with DAL still stay
in memory and are not collected by gc.
Please advise so I could import all 1000 parts with constant memory usage.
Best Regards
Lucas
