sorry ... i don't know what it means to batch writes? Are you referring to the upload tool or something else? Do batched writes cost less? Can you refer me to the documentation page on it? Thanks so much.
On Feb 21, 11:50 am, stevep <[email protected]> wrote: > You are batching those writes if it is feasible, right? -stevep > > On Feb 21, 9:56 am, James Gilliam <[email protected]> wrote: > > > > > > > > > Yes ... I saw the 5 per sec in the admin ... > > > Also, I just realized why I think gae created so many instances when i > > had the queue at 1/s ... I had assumed the tasks would take much less > > than a second (which was true at the beginning); but as the number of > > entity writes continued, the latency for the writes got larger and > > eventually took longer than a second ... at which point gae started > > new instances to keep up with the 1 sec rate ... i had assumed it > > would not consider starting a new task until the previous one in the > > queue finished -- this appears to not be the semantics > > > My solution is to slow down the tasks even further ... to say 15 / > > min ... so that new instances will not be created ... > > > On Feb 19, 12:21 pm, Nicholas Verne <[email protected]> wrote: > > > > James, > > > > The misreporting of number of tasks in a queue is a known bug that > > > we're working on. It causes undue alarm for our users. > > > > How do you know that GAE changed the target rate to 5 / sec ? Did you > > > observe this in the admin console? > > > > The queue's rate is the rate at which tasks are dispatched to your > > > app. If your queue is set to run 1 / sec and your task latency is > > > longer than 1 second, you should expect to have more instances spin > > > up. > > > > If you wish to prevent more than one task running at a time, set your > > > queue's max_concurrent_requests to 1. > > > > Nick Verne > > > > On Mon, Feb 20, 2012 at 3:35 AM, James Gilliam <[email protected]> > > > wrote: > > > > I am using task queue in python. > > > > > I added 10,000 tasks for a queue set to run at 1/sec and a bucket of > > > > 1 ... they were short tasks and I didn't want them to run > > > > immediately ... I didn't even want them to run multiples per second > > > > even though the latency was low ... it was ok if it took hours as I > > > > knew it would at 1/sec. I figured it would create a second instance > > > > as the site was continuing to run ... but I was pretty surprised with > > > > what actually happen. > > > > > At first the queue started processing fine ... approx 60 tasks running > > > > per minute ... > > > > > Then GAE decided to run them at a faster rate, change the target rate > > > > to 5 / sec and created lots of new instances ... > > > > > When there were about 3000 tasks left ... GAE went crazy and said > > > > there were nearly 200,000 tasks in the queue ... > > > > > As far as I could tell, the number of tasks being reported was just an > > > > anomaly ... but it was alarming given that GAE was creating instances > > > > like crazy. > > > > > The tasks would add new records (sometimes) and I noticed that the > > > > latency of these tasks increased the longer the tasks ran. At first > > > > just 200ms ... by the end nearly 1400 ms. > > > > > So -- > > > > > 1. The most important question -- Why did GAE not respect my desire to > > > > run the queue at 1 / sec ? Is there a way I can stop GAE from > > > > overriding my wishes? > > > > > 2. Was the latency increasing because the records being added were > > > > slowing new records from being added? > > > > > 3. The invalid number of tasks seems like a harmless bug since new > > > > phantom tasks were not actually created (as far as I know). > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "Google App Engine" group. > > > > To post to this group, send email to [email protected]. > > > > To unsubscribe from this group, send email to > > > > [email protected]. > > > > For more options, visit this group > > > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
