Thanks a lot for your valuable replies! I'll have to check out the current state of the map reduce lib as I remember from Google IO that it does not support certain filters etc. Simple question though: What is the maximum of updated entities/minute inside a transaction that you have seen in the real world?
On 29 Nov., 18:04, Eli Jones <[email protected]> wrote: > You mention that "tasks get rescheduled for some reason".. what is the > reason? Does this reason occur frequently? > > Also, there is no way to evaluate how fast you can perform your > modifications since you haven't shown the code that you are currently using. > > There may be several simple tweaks to your existing code that could make it > much faster. > > > > > > > > On Mon, Nov 29, 2010 at 9:29 AM, dflorey <[email protected]> wrote: > > Thanks for your response. I though that mapreduce will also sit on top > > of task queue and will most likely give any speed improvements over my > > approach? > > I am seeing ~1500 tasks per minute getting executed. Will mapreduce > > give higher numbers? > > > Daniel > > > On 29 Nov., 10:41, Peter Ondruska <[email protected]> wrote: > > > I would you mapreduce for GAE, seehttp:// > > code.google.com/p/appengine-mapreduce/. > > > It has been integrated with latest SDK so no need to download, I use > > > it with Python--just make sure to import > > > google.appengine.ext.mapreduce. > > > > On 29 lis, 10:06, dflorey <[email protected]> wrote: > > > > > Hi, > > > > I'm looking for the most effective way to update 50000 entities + one > > > > of the child entities each. > > > > Right now I'm using a task per transaction to be able to modify the > > > > entity and the child entities inside a transaction to make the task > > > > idempotent. > > > > I'm using sharded counters to check when the operation is done. > > > > Everything works fine, but it takes very long (=minutes to hours) to > > > > perform the modifications. > > > > I'm getting no concurrent modification exceptions etc. at all, but > > > > tasks get rescheduled for some reason and wait for a long time before > > > > getting executed depending on the number of retries. > > > > > Is there a way to speed things up? > > > > I'm looking for a solution that will execute the update almost > > > > immediately :-) > > > > My tasks take less than 1000ms each and I can see ~30 instances in the > > > > dashboard. > > > > > Thanks for any ideas, > > > > > Daniel > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%2Bunsubscrib > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
