Start a another datastore that doesn't use the same entity, but uses an incremental system.
On Write: In a table with format Writes to do, Entity to Overwrite, Data for Entity 1,Brandon Wirtz, +$120 2,Brandon Wirtz, -$120 3,Brandon Wirtz, +$0.66 Increment Write to do count, and store transaction. If a Read request for the entity comes in: Get Entity Get Latest committed "Write To Do Value" Check Write Cache Recalculate Value based on Entity Value and All Changes On Batch: For batch size calculate all values, and update entities Store Late todo value as Last committed write. You can now batch as often as every 2 seconds and not hit the entity write limit. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Nikolaj Sent: Thursday, September 22, 2011 3:59 PM To: Google App Engine Subject: [google-appengine] How to throttle or batch transactional updates to an entity group Hi there, I'm developing an application that requires me to create lots of child entities within the same entity group transactionally. The tasks being generated to do this come in unpredictable bursts - sometimes 100/s for an entity group. This is clearly exceeding the 1/s limit for transaction contention and causing some real problems for my poor application. I need a way to either batch the required updates for each entity group so I can run through them serially or otherwise schedule each update to run on a 1/s basis. The ideas I've come up with so far: 1. Use some kind of memcache'd timestamp or counter to schedule or throttle the updates. My question is how to do this in a way that will work with 100's of tasks hungrily looking for their time slot? Should I do a CAS for when the last task was scheduled (would this even work with the contention we're talking about?) or can I use an atomic operation like incr() somehow? 2. Store the work items somewhere and then at some later time query for the work items for an entity group and create the children in serial. I'd like to avoid this, if possible, as it creates many other headaches. My application is already using the taskqueue to great effect. This really seems like a queue problem - if only there was some kind of dynamic subqueue where I could add tasks with a 'namespace' of the entity group. Then entity groups could run in parallel, but the entity group items would still be in serial. Does something like that exist? Any suggestions or experiences on how I can work around this would be appreciated. Many thanks, Nikolaj -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
