Hi James, Look at item 5: http://googleappengine.blogspot.com/2009/06/10-things-you-probably-didnt-know-about.html
Cost is the same: http://code.google.com/appengine/docs/python/datastore/entities.html Batching entity puts may significantly decrease your latency if it fits what you are doing. You may want to send the batch put off to a task queue using a deferred function because I believe the cumulative probability of a write failure is higher with batch puts and you can leverage deferred's retry feature (same as task queues). However, you will need to pay attention to the 1 meg payload limit if you are deferring very large numbers of puts. Test the 1 meg limit by saving to memcache to see if it errors out. HTH -stevep On Feb 21, 12:00 pm, James Gilliam <[email protected]> wrote: > sorry ... i don't know what it means to batch writes? Are you > referring to the upload tool or something else? Do batched writes > cost less? Can you refer me to the documentation page on it? Thanks > so much. > > On Feb 21, 11:50 am, stevep <[email protected]> wrote: > > > > > > > > > You are batching those writes if it is feasible, right? -stevep > > > On Feb 21, 9:56 am, James Gilliam <[email protected]> wrote: > > > > Yes ... I saw the 5 per sec in the admin ... > > > > Also, I just realized why I think gae created so many instances when i > > > had the queue at 1/s ... I had assumed the tasks would take much less > > > than a second (which was true at the beginning); but as the number of > > > entity writes continued, the latency for the writes got larger and > > > eventually took longer than a second ... at which point gae started > > > new instances to keep up with the 1 sec rate ... i had assumed it > > > would not consider starting a new task until the previous one in the > > > queue finished -- this appears to not be the semantics > > > > My solution is to slow down the tasks even further ... to say 15 / > > > min ... so that new instances will not be created ... > > > > On Feb 19, 12:21 pm, Nicholas Verne <[email protected]> wrote: > > > > > James, > > > > > The misreporting of number of tasks in a queue is a known bug that > > > > we're working on. It causes undue alarm for our users. > > > > > How do you know that GAE changed the target rate to 5 / sec ? Did you > > > > observe this in the admin console? > > > > > The queue's rate is the rate at which tasks are dispatched to your > > > > app. If your queue is set to run 1 / sec and your task latency is > > > > longer than 1 second, you should expect to have more instances spin > > > > up. > > > > > If you wish to prevent more than one task running at a time, set your > > > > queue's max_concurrent_requests to 1. > > > > > Nick Verne > > > > > On Mon, Feb 20, 2012 at 3:35 AM, James Gilliam <[email protected]> > > > > wrote: > > > > > I am using task queue in python. > > > > > > I added 10,000 tasks for a queue set to run at 1/sec and a bucket of > > > > > 1 ... they were short tasks and I didn't want them to run > > > > > immediately ... I didn't even want them to run multiples per second > > > > > even though the latency was low ... it was ok if it took hours as I > > > > > knew it would at 1/sec. I figured it would create a second instance > > > > > as the site was continuing to run ... but I was pretty surprised with > > > > > what actually happen. > > > > > > At first the queue started processing fine ... approx 60 tasks running > > > > > per minute ... > > > > > > Then GAE decided to run them at a faster rate, change the target rate > > > > > to 5 / sec and created lots of new instances ... > > > > > > When there were about 3000 tasks left ... GAE went crazy and said > > > > > there were nearly 200,000 tasks in the queue ... > > > > > > As far as I could tell, the number of tasks being reported was just an > > > > > anomaly ... but it was alarming given that GAE was creating instances > > > > > like crazy. > > > > > > The tasks would add new records (sometimes) and I noticed that the > > > > > latency of these tasks increased the longer the tasks ran. At first > > > > > just 200ms ... by the end nearly 1400 ms. > > > > > > So -- > > > > > > 1. The most important question -- Why did GAE not respect my desire to > > > > > run the queue at 1 / sec ? Is there a way I can stop GAE from > > > > > overriding my wishes? > > > > > > 2. Was the latency increasing because the records being added were > > > > > slowing new records from being added? > > > > > > 3. The invalid number of tasks seems like a harmless bug since new > > > > > phantom tasks were not actually created (as far as I know). > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "Google App Engine" group. > > > > > To post to this group, send email to > > > > > [email protected]. > > > > > To unsubscribe from this group, send email to > > > > > [email protected]. > > > > > For more options, visit this group > > > > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
