Re: django cron job - stops after reading some portion of huge file - why is this?

2013-10-15 Thread Bill Freeman
Yes, you should split the db activity into sensible transactions, since information about how to roll back is being stored somewhere (though some DBs may not have a problem with this. You've added a whole new dimension when you say that this data is not, in fact, a local file that you are

Re: django cron job - stops after reading some portion of huge file - why is this?

2013-10-15 Thread doniyor
Awesome, then let me try these things you mentioned.. i let you know then.. thanks in tons for now On Tuesday, October 15, 2013 5:04:02 PM UTC+2, ke1g wrote: > > Yes, you should split the db activity into sensible transactions, since > information about how to roll back is being stored

Re: django cron job - stops after reading some portion of huge file - why is this?

2013-10-15 Thread doniyor
yes, db code is doin all these calls in single transaction, i mean, i am not using transactions, may be this is the reason? this is my cron code: http://pastebin.com/Lrym1z8E i know, very ugly code, it is saving at least some objects into db this is url where i am reading my xml file from:

Re: django cron job - stops after reading some portion of huge file - why is this?

2013-10-15 Thread Bill Freeman
One possibility is that your code keeps all that is read (or something derived from it) in memory, and you are running out. E.g.; Is your database code trying to do all this in a single transaction? Another possibility is that something in the file at that spot triggers a but in your code that

django cron job - stops after reading some portion of huge file - why is this?

2013-10-15 Thread doniyor
I am reading file from url and parsing it and saving some information out of this file into db - using cron job. i am testing now in my local dev. the problem is: job is reading file and saving into db without any problem but after some time, since file is very huge approx. >8GB, job doesnot