I presume that each of these items will be persisted to the datastore in a separate entity group. If not, then there is a limit as to the rate at which updates to a given entity group can be made (from my recollection of the documentation: of the order of 1 to 5 times per second); this limit would affect your design decision.
On Aug 10, 1:11 pm, Pieter Coucke <[email protected]> wrote: > Hi, > > I'm calling a webservice which returns an XML with 100 items. This > webservice is called many times a second. > > I want to save the items as fast as possible and so I was thinking of > creating a task for each item (with the serialized item object as task > payload). Each such task does a select on the datastore to know if the item > already exists and if it doesn't exist, inserts a new entity (or else update > the entity). > > I suppose adding a task also has overhead but still is faster than calling > the datastore. > > I'd like to ask if any of you has considered this approach or has some > performance numbers or advice about which is best (fastest): > - perform all 100 selects and updates in one task (here the 30 second > execution limit is dangerous) > - one task for each item insert (since the queue call rate is limited to 50, > the "one task per item" approach will take almost 2 seconds) > - one task for each item insert where the tasks are round-robin added to 5 > queues so the call rate is increased to 250 / second > - 10 tasks with each 10 inserts (here the problem can become the task > payload which is limited to 10 KB) > > Or should I ask for a queue call rate increase (don't think this is > possible)? > > Thanks, > > Pieter Coucke -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
