For a similar problem I use a task A which fetches 10 entities, creates subtasks B for them and then creates a new task A with the datastore cursor as a parameter so this task can continue creating subtasks.
On Mon, Jul 12, 2010 at 1:31 AM, Phil McDonnell <[email protected]>wrote: > I know there are a number of different frameworks and mechanisms to process > data offline on GAE. I need to have a daily job that scans one of my > datastore tables and computes an updated profile for all my users. What > would be the best mechanism to do this? Are task queues appropriate? > Should I be using some sort of mapreduce? If I should use some sort of > mapreduce framework, how does that integrate with GAE? My user base right > now is small, but I intend to scale this. > > Thanks, > Phil > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Pieter Coucke Onthoo BVBA http://www.onthoo.com http://www.koopjeszoeker.be -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
