The daily limit on Task Queue API calls is 20,000,000 and the limit on URL fetch is 32,000 calls/minute. So at the rate your suggesting your going to run into some walls. I'd say you'd have to request an increase. I'd get the okay from GAE before going down this road.
On Fri, Jan 14, 2011 at 1:22 PM, supercobra <[email protected]> wrote: > Hi Eli, > > Thanks for the great feedback. If Task Queue Java is as evolved as > Phyton that maybe the way to go. > > > -- [email protected] > http://supercobrablogger.blogspot.com/ > > > > On Fri, Jan 14, 2011 at 12:52 PM, Eli Jones <[email protected]> wrote: > > You should probably start experimenting with the Taskqueue to get a > better > > idea how it functions and what is possible. > > When you create a Task.. you can define a countdown or eta value.. so, > the > > task gets added.. but it is scheduled to execute at a future time (not > > immediately). > > So, you can just create a named, chained task that does a batch of > > URLfetches asynchronously with a callback (not sure how many can be done > at > > once, one way to find out is to test it.).. and once it is done gathering > > and putting data.. it schedules the next 5 minute batch. > > > http://code.google.com/appengine/docs/python/urlfetch/asynchronousrequests.html > > See Nick Johnson for an overview of how to do chaining with deferred (I > > personally prefer regular Taskqueue, but deferred has its positive > aspects): > > http://blog.notdot.net/2010/03/Task-Queue-task-chaining-done-right > > You could also have a cron job that creates all of your scheduled tasks > for > > the day at once. Depending on if there would be no issues with some > tasks > > not finishing before the next 5 minute task. > > > > On Fri, Jan 14, 2011 at 1:24 PM, supercobra <[email protected]> > wrote: > >> > >> Hello again, > >> > >> > >> > Again as Eli mentioned, to offer any solutions we'd need to know what > >> > you're trying to do in all those tasks. > >> > >> We are exploring the idea of migrating our website monitoring service > >> Montastic (www.montastic.com) to GAE. Right now it works on MySQL but > >> scalability becomes challenging and we need to explore a > >> re-architecture of our system. GAE would be ideal if it was possible. > >> So we do need to do a LOT of UrlFetch-es every 1, 2, 5 minutes. > >> > >> One of the challenge is to wait for 5 minutes. E.g. Fetch a URL, store > >> results, wait 5 min, do it again. Since a queue will execute the task > >> almost immediately (if it is empty) this would not work unless the > >> queue is filled w/ a known number of tasks. > >> > >> > >> Any suggestion welcome. > >> > >> > >> Daniel > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<google-appengine%[email protected]> > . > >> For more options, visit this group at > >> http://groups.google.com/group/google-appengine?hl=en. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%[email protected]> > . > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
