yes, db code is doin all these calls in single transaction, i mean, i am not using transactions, may be this is the reason?
this is my cron code: http://pastebin.com/Lrym1z8E i know, very ugly code, it is saving at least some objects into db this is url where i am reading my xml file from: http://www.bookatable.com/feed.aspx?channel=524&secret=10F70A84-76D7-C56C-2350-779A1A849819&language=de-DE&type=restaurants also i noticed now that in db, there are objects whose some fields are not fully filled out even if the xml file does have those information. it means, this is a transaction issue, right? could you please take a look at the code? would transaction solve this issue? On Tuesday, October 15, 2013 3:40:42 PM UTC+2, ke1g wrote: > > One possibility is that your code keeps all that is read (or something > derived from it) in memory, and you are running out. > > E.g.; Is your database code trying to do all this in a single transaction? > > Another possibility is that something in the file at that spot triggers a > but in your code that contains an infinite loop. > > There are other possibilities. But there's no diagnosing it with the > information you've given. > > Can you, in python, read through the file, doing nothing with the data? > E.g.: > > f = open('your/file/path/here') > n = 0 > s = True > while s: > s = f.read(1024*1024) > n += len(s) > print n > print 'done' > > That should work. If not, does your O/S not correctly handle files that > big? > > Bill > > > On Tue, Oct 15, 2013 at 6:55 AM, doniyor <[email protected] <javascript:> > > wrote: > >> I am reading file from url and parsing it and saving some information out >> of this file into db - using cron job. >> >> i am testing now in my local dev. >> >> the problem is: job is reading file and saving into db without any >> problem but after some time, since file is very huge approx. >8GB, job >> doesnot do anything and freezes, without giving any error, >> >> i am using django 1.4, python 2.7 and postgresql. is there any limit for >> writing into db? why is it freezing? >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/b567f273-508b-4f14-9d40-197af8c8a079%40googlegroups.com >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b61fec61-7481-4113-ab8c-31b7143df3f5%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
