James, The misreporting of number of tasks in a queue is a known bug that we're working on. It causes undue alarm for our users.
How do you know that GAE changed the target rate to 5 / sec ? Did you observe this in the admin console? The queue's rate is the rate at which tasks are dispatched to your app. If your queue is set to run 1 / sec and your task latency is longer than 1 second, you should expect to have more instances spin up. If you wish to prevent more than one task running at a time, set your queue's max_concurrent_requests to 1. Nick Verne On Mon, Feb 20, 2012 at 3:35 AM, James Gilliam <[email protected]> wrote: > I am using task queue in python. > > I added 10,000 tasks for a queue set to run at 1/sec and a bucket of > 1 ... they were short tasks and I didn't want them to run > immediately ... I didn't even want them to run multiples per second > even though the latency was low ... it was ok if it took hours as I > knew it would at 1/sec. I figured it would create a second instance > as the site was continuing to run ... but I was pretty surprised with > what actually happen. > > At first the queue started processing fine ... approx 60 tasks running > per minute ... > > Then GAE decided to run them at a faster rate, change the target rate > to 5 / sec and created lots of new instances ... > > When there were about 3000 tasks left ... GAE went crazy and said > there were nearly 200,000 tasks in the queue ... > > As far as I could tell, the number of tasks being reported was just an > anomaly ... but it was alarming given that GAE was creating instances > like crazy. > > The tasks would add new records (sometimes) and I noticed that the > latency of these tasks increased the longer the tasks ran. At first > just 200ms ... by the end nearly 1400 ms. > > So -- > > 1. The most important question -- Why did GAE not respect my desire to > run the queue at 1 / sec ? Is there a way I can stop GAE from > overriding my wishes? > > 2. Was the latency increasing because the records being added were > slowing new records from being added? > > 3. The invalid number of tasks seems like a harmless bug since new > phantom tasks were not actually created (as far as I know). > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
