I am using task queue in python. I added 10,000 tasks for a queue set to run at 1/sec and a bucket of 1 ... they were short tasks and I didn't want them to run immediately ... I didn't even want them to run multiples per second even though the latency was low ... it was ok if it took hours as I knew it would at 1/sec. I figured it would create a second instance as the site was continuing to run ... but I was pretty surprised with what actually happen.
At first the queue started processing fine ... approx 60 tasks running per minute ... Then GAE decided to run them at a faster rate, change the target rate to 5 / sec and created lots of new instances ... When there were about 3000 tasks left ... GAE went crazy and said there were nearly 200,000 tasks in the queue ... As far as I could tell, the number of tasks being reported was just an anomaly ... but it was alarming given that GAE was creating instances like crazy. The tasks would add new records (sometimes) and I noticed that the latency of these tasks increased the longer the tasks ran. At first just 200ms ... by the end nearly 1400 ms. So -- 1. The most important question -- Why did GAE not respect my desire to run the queue at 1 / sec ? Is there a way I can stop GAE from overriding my wishes? 2. Was the latency increasing because the records being added were slowing new records from being added? 3. The invalid number of tasks seems like a harmless bug since new phantom tasks were not actually created (as far as I know). -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
