Hi Sarfaraz, Which sort of deadline exceeded error are you getting? If it's from the URLFetch call, you can increase the deadline; on offline requests (task queue and cron job) that can be up to 10 minutes.
You can generally ignore the 'too much CPU' warning; there are no longer per-handler or per-page quotas, and offline requests may use as much CPU as they need (providing you have the quota for it!). You can improve the efficiency of your code, though: Instead of calling .put() in each iteration of the loop, batch the results up and store them with a single put call: entities = [] for row in rows: q = Quotes(key_name=cells[0]) # Fill in q, or pass the values as arguments to the constructor above entities.append(q) db.put(entities) The other modification, above, is to simply call the constructor instead of get_or_insert. get_or_insert executes a transaction to look for an existing entity, and insert one if it doesn't exist. You're simply overwriting all the values of the entity, though, so there's no need to fetch the old one or do it transactionally - you can simply create a new entity, overwriting any old data. -Nick Johnson On Thu, Apr 7, 2011 at 7:00 AM, Sarfaraz <[email protected]>wrote: > Hi, > I am running a cron every 5 minutes to get csv data (stock prices) from a > url using urllib2. After splitting the data by Newline and Comma I am > storing the data in datastore > > I am getting many deadlineexceed errors and warning messages that the cron > "uri > uses a high amount of cpu and may soon exceed its quota" > > I am attaching a few snapshots from my dashboard, and also pasting my code > below for a review: is this Normal or I am doing somethign wrong. > > > *Model* > > > class Quotes(db.Model): #Symbol is stored as the key_name > PriceDate = db.StringProperty() # I am using string here because i just > show this value no calculations or sorting on this ignore it. > Open = db.FloatProperty() > High = db.FloatProperty() > Low = db.FloatProperty() > Last = db.FloatProperty() > Change = db.FloatProperty() > PerChange = db.FloatProperty() > PrevClose = db.FloatProperty() > Vol = db.IntegerProperty() > Val = db.IntegerProperty() > UpdatedOn = db.DateTimeProperty(auto_now=True) > > ** > *Code for URL fetch, parsing and storing into datastore:* > > I get the csv data first which consists of 147 Rows and 9 Columns (comma > seperated values) rows may increase in future > First I split by New Line Charachter to get all Rows which are approx 147 > Then I loop through each row and split it by comma to get column values > (approx 9 columns) > while looping for the rows I fetch an entity by using " get_or_insert ( > key_name ) > then I update all the properties and call the put(). > > Please check the detailed code below ( I have removed error handling and > logging stuff for brevity) > > These are a few lines from the csv data I recieve from the url. > *********************************************************************** > 1010,4/6/2011 3:32:01 PM,26.00,26.20,26.00,26.20,0.10,207359,5423720 > 1020,4/6/2011 3:32:01 PM,19.25,19.60,19.10,19.60,0.35,739595,14399067 > 1030,4/6/2011 3:32:01 PM,20.20,20.30,20.15,20.30,0.10,31833,643936 > 1040,4/6/2011 3:32:01 PM,30.40,30.80,30.40,30.80,0.00,10621,325830 > 1050,4/6/2011 3:32:01 PM,49.30,50.50,49.00,50.00,1.20,126361,6326252 > > *********************************************************************** > > HERE IS THE CODE > > url = "http://www.example.com/somefile.ashx"; > > result = urllib2.urlopen(url) > result = result.read() > rows = result.split("\n") > for row in rows: > cells = row.split(",") > q = Quotes.get_or_insert( (str(cells[0])).strip() ) ## cells[0] > contains my key_name > q.PriceDate = cells[1] > q.Open = float(cells[2]) #opening price > q.High = float(cells[3]) #days high price > q.Low = float(cells[4]) #days low price > q.Last = float(cells[5]) > q.Change = float(cells[6]) > q.PrevClose = float(q.Last - q.Change) > q.PerChange = round( (q.Change * 100 ) / q.PrevClose , 2) > q.Vol = int(cells[7]) > q.Val = int(cells[8]) > q.put() > ** > ** > *This is using too much cpu and exceeding deadline because I am fetching 1 > entity at a time and using put() for each entity thats 147 times ? AND > also because I am doing some calculations such as q.PrevClose = > (q.Last-q.Change) instead of calculating it in a diffrent variable and then > assigning it to property. are these the reason ? please help me in getting > this right* > ** > *Attached: Dashbaord Screnshots* > ** > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
