You should probably start experimenting with the Taskqueue to get a better idea how it functions and what is possible.
When you create a Task.. you can define a countdown or eta value.. so, the task gets added.. but it is scheduled to execute at a future time (not immediately). So, you can just create a named, chained task that does a batch of URLfetches asynchronously with a callback (not sure how many can be done at once, one way to find out is to test it.).. and once it is done gathering and putting data.. it schedules the next 5 minute batch. http://code.google.com/appengine/docs/python/urlfetch/asynchronousrequests.html See Nick Johnson for an overview of how to do chaining with deferred (I personally prefer regular Taskqueue, but deferred has its positive aspects): http://blog.notdot.net/2010/03/Task-Queue-task-chaining-done-right You could also have a cron job that creates all of your scheduled tasks for the day at once. Depending on if there would be no issues with some tasks not finishing before the next 5 minute task. On Fri, Jan 14, 2011 at 1:24 PM, supercobra <[email protected]> wrote: > Hello again, > > > > Again as Eli mentioned, to offer any solutions we'd need to know what > > you're trying to do in all those tasks. > > We are exploring the idea of migrating our website monitoring service > Montastic (www.montastic.com) to GAE. Right now it works on MySQL but > scalability becomes challenging and we need to explore a > re-architecture of our system. GAE would be ideal if it was possible. > So we do need to do a LOT of UrlFetch-es every 1, 2, 5 minutes. > > One of the challenge is to wait for 5 minutes. E.g. Fetch a URL, store > results, wait 5 min, do it again. Since a queue will execute the task > almost immediately (if it is empty) this would not work unless the > queue is filled w/ a known number of tasks. > > > Any suggestion welcome. > > > Daniel > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
