Hi Daniel, Map reduce could speed things up. You say your using one task per transaction, so there is probably some overhead and delay in executing those whereas map reduce will call your map function for several different entities within one task. It keeps track of how much time has elapsed and will keep executing the map function for different entities until some threshold has been reached. Unfortunately it seems you must execute your updates within a transaction. If you could go without that then you'd get a huge boost in performance I believe. That is because you can add your updates to the DatastoreMutationPool object which will batch put them once that tasks threshold has been reached. Stephen
On Mon, Nov 29, 2010 at 7:29 AM, dflorey <[email protected]> wrote: > Thanks for your response. I though that mapreduce will also sit on top > of task queue and will most likely give any speed improvements over my > approach? > I am seeing ~1500 tasks per minute getting executed. Will mapreduce > give higher numbers? > > Daniel > > On 29 Nov., 10:41, Peter Ondruska <[email protected]> wrote: > > I would you mapreduce for GAE, seehttp:// > code.google.com/p/appengine-mapreduce/. > > It has been integrated with latest SDK so no need to download, I use > > it with Python--just make sure to import > > google.appengine.ext.mapreduce. > > > > On 29 lis, 10:06, dflorey <[email protected]> wrote: > > > > > > > > > > > > > > > > > Hi, > > > I'm looking for the most effective way to update 50000 entities + one > > > of the child entities each. > > > Right now I'm using a task per transaction to be able to modify the > > > entity and the child entities inside a transaction to make the task > > > idempotent. > > > I'm using sharded counters to check when the operation is done. > > > Everything works fine, but it takes very long (=minutes to hours) to > > > perform the modifications. > > > I'm getting no concurrent modification exceptions etc. at all, but > > > tasks get rescheduled for some reason and wait for a long time before > > > getting executed depending on the number of retries. > > > > > Is there a way to speed things up? > > > I'm looking for a solution that will execute the update almost > > > immediately :-) > > > My tasks take less than 1000ms each and I can see ~30 instances in the > > > dashboard. > > > > > Thanks for any ideas, > > > > > Daniel > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
