Yes, you can. The Mapper API has been designed for that kind of batch processing, especially for the type where the large amount of execution time is caused by the large amount of individual entities to process (instead of long execution time for each of the entities).
I recommend having a look at http://googleappengine.blogspot.com/2010/07/introducing-mapper-api.html and other links mentioned on that page, including the slides of the presentation about the concept, Ikai's post about it, the project home page and the Getting started's. The library is currently early experimental and clearly there is room for improvement, but on the other hand improvements are being made; we had an issue with very unbalanced shard distribution, which got fixed by means of the __scatter__ property. Are you using Python or Java? You might want to have a look at http://code.google.com/p/appengine-mapreduce/issues/detail?id=41 In case you are using Java you can fix it yourself and contribute to project. The source code is available and not really rocket science. Erwin On 21 jan, 18:57, Mayumi Liyanage <[email protected]> wrote: > Hi, I need to iterate over all the entities in the Datastore and send > out emails once a day asynchronously to the actual app. > Usual way to do this would be to invoke a Servlet using cron which > would iterate over all the entities to send emails out. However, our > data is growing at the rapid rate and sooner or later we will have a > issue with 30 sec limit problem. > What would be the best way to do this operation using app engine > without worrying about 30 sec limit? > > Can we do above using the Mapper API? If so how can we invoke a mapper > from the servlet? > > Thanks. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
