Linux3.0.0-17-generic #30-Ubuntu SMP Thu Mar 8 20:45:39 UTC 2012 x86_64
x86_64 x86_64 GNU/Linux
Python 2.7.2+ (default, Oct 4 2011, 20:06:09)
[GCC 4.6.1] on linux2
I used gae tasks as alternative to loops so i create task for each file
import. I don't know if it will cause the same result on gae but on local
env memory after task execution was not released for web2py server process.
def csv_import():
imp = db[request.vars.table].import_from_csv_file
file = open(request.vars.filename, 'r')
imp(file)
db._timings = []
file.close()
return True
def tasks():
path = os.path.join(os.path.join(request.folder,'uploads',
request.vars.table))
from google.appengine.api import taskqueue
files = os.listdir(path)
for file in files:
taskqueue.add(url = URL(r=request, c='import',f='csv_import', vars
= {'table': request.vars.table, 'filename': os.path.join(path, file)}),
method='GET')
return "Tasks added"
W dniu poniedziałek, 9 kwietnia 2012 14:26:36 UTC+2 użytkownik Massimo Di
Pierro napisał:
>
> What os? what python version?
>
> On Sunday, 8 April 2012 16:08:00 UTC-5, Czeski wrote:
>>
>> Hi,
>>
>> I am new web2py user and I have some performance problems with
>> import_from_csv_file method. First of all i have big collection of data
>> that i want to upload to Google App Engine. I splited data into 1000 parts,
>> each contains csv serialized rows - about 1367 rows per file. I am doing
>> loop to import each file to database using:
>>
>> def csv_import():
>>
>> for i in xrange(0, 1000):
>> file = open(os.path.join(request.folder,'private', 'geonames',
>> 'chunk_' + str(i)), 'r')
>> db.geonames.import_from_csv_file(file)
>> db._timings = []
>> file.close()
>>
>> As You can see it is rather simple method to achieve this. But the main
>> problem is that every loop iteration is increasing the overall memory usage
>> for application. It is never stoped and in 10 iteration it used all system
>> resources and app is terminated.
>>
>> I think that with every iteration some objects related with DAL still
>> stay in memory and are not collected by gc.
>>
>> Please advise so I could import all 1000 parts with constant memory usage.
>>
>> Best Regards
>> Lucas
>>
>
