I think Eli has a good suggestion (again), use task-chaining with countdowns + async urlfetches in small batches. Just beware, countdowns are only an estimate, and if the queue is backing up the task may not run when you want.
Just thinking about this, I would probably try to batch similarly performing websites into small batches to monitor together. So if you got sites that typically respond very fast group them, like-wise for slow sites. I suspect that will help you optimize your queue layouts, maybe you could use some queues for 'fast' and others for 'slow' groups. Just some thoughts. I also agree with some of the other commenters, you should setup some tests and see if you still feel like this is the right platform for your app. Robert On Sat, Jan 15, 2011 at 11:03, supercobra <[email protected]> wrote: > The countdown parameter of TaskQueue is indeed a big help here. Thanks > for pointing that out. > > -- [email protected] > http://supercobrablogger.blogspot.com/ > > > > On Fri, Jan 14, 2011 at 3:41 PM, Uros Trebec <[email protected]> wrote: >> re >> >> On Jan 14, 7:24 pm, supercobra <[email protected]> wrote: >>> One of the challenge is to wait for 5 minutes. E.g. Fetch a URL, store >>> results, wait 5 min, do it again. Since a queue will execute the task >>> almost immediately (if it is empty) this would not work unless the >>> queue is filled w/ a known number of tasks. >>> >>> Any suggestion welcome. >> >> You can use the 'countdown' parameter in Task constructor ( >> http://code.google.com/appengine/docs/python/taskqueue/tasks.html#Task >> ) to set the number of seconds for the Task to wait in the queue >> before executing. I use this for scheduling a task a few minutes in >> the future when UrlFetch returns the data I already have. >> >> lp, >> Uros >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
