Thanks for your response. I though that mapreduce will also sit on top of task queue and will most likely give any speed improvements over my approach? I am seeing ~1500 tasks per minute getting executed. Will mapreduce give higher numbers?
Daniel On 29 Nov., 10:41, Peter Ondruska <[email protected]> wrote: > I would you mapreduce for GAE, > seehttp://code.google.com/p/appengine-mapreduce/. > It has been integrated with latest SDK so no need to download, I use > it with Python--just make sure to import > google.appengine.ext.mapreduce. > > On 29 lis, 10:06, dflorey <[email protected]> wrote: > > > > > > > > > Hi, > > I'm looking for the most effective way to update 50000 entities + one > > of the child entities each. > > Right now I'm using a task per transaction to be able to modify the > > entity and the child entities inside a transaction to make the task > > idempotent. > > I'm using sharded counters to check when the operation is done. > > Everything works fine, but it takes very long (=minutes to hours) to > > perform the modifications. > > I'm getting no concurrent modification exceptions etc. at all, but > > tasks get rescheduled for some reason and wait for a long time before > > getting executed depending on the number of retries. > > > Is there a way to speed things up? > > I'm looking for a solution that will execute the update almost > > immediately :-) > > My tasks take less than 1000ms each and I can see ~30 instances in the > > dashboard. > > > Thanks for any ideas, > > > Daniel -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
