With a little thought, you shouldn't encounter an issue enqueuing the tasks. That said, as Eli mentioned, you would need to use multiple queues with high exec rates to get anything near that type of throughput. You'll need to run the tasks at a rate of about 334 / second; however, I strongly doubt you are going to get close to that with a 5,000ms task. Your app will probably not be given enough instances. If you can get the task latency down to a few hundred ms / task, I suspect you would have a chance of getting it to work.
Again as Eli mentioned, to offer any solutions we'd need to know what you're trying to do in all those tasks. Robert On Thu, Jan 13, 2011 at 19:01, Eli Jones <[email protected]> wrote: > You couldn't even execute 100,000 tasks in a 5 minute time frame with the > ideal 100/s execution rate. > Doing 100 tasks a second means it would take 16 minutes just to start those > 100,000 tasks. > Now, you might could create multiple queues.. but I'm not sure what the > absolute tasks per second execution rate is. > > So, maybe you could group the activities into groups of 100.. so you only > needed 1,000 recurring tasks. > Then you could chain each individual task to re-add itself to the taskqueue > with an ETA at the next 5 minute mark after it finished its current work. > Though, any solutions will depend on what exactly these recurring activities > are doing.. is it a URLfetch? or an email? or just datastore gets and puts? > > On Thu, Jan 13, 2011 at 6:19 PM, Supercobra Thatbytes <[email protected]> > wrote: >> >> Hello there, >> >> I am looking for ideas. >> >> I want to perform 100000 recurring activities every 5 minutes. Each >> activity takes about 5 seconds. >> >> I think using 5-min cron to start 100k+ jobs would not be cpu/cost >> effective. >> >> Any ideas on how to do this w/ the least overhead? >> >> Daniel >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
