this is exactly what I'm seeing with the Task Queue as of yesterday. it was ripping fast 2 days ago, I was getting close to 150 features/ second upload/index. now this has dropped considerably and there are many timeouts on the TQ after only 10secs (startup time, not even hitting my code).
http://groups.google.com/group/google-appengine/browse_thread/thread/85dc115fac469b21# On Apr 16, 10:20 am, hawkett <[email protected]> wrote: > Hi Nick, > > This is a little disconcerting - would it be fair to say that if I > were using the pattern you documented > herehttp://code.google.com/appengine/articles/deferred.html, > then my latencies would not be low (since I am pushing the > DeadlineExceededError on every deferred operation), and my app will > become request limited? Perhaps this pattern should be processing only > one or two records at a time (and then risking the task limit)? > Perhaps it does not apply to requests initiated from the task queue? > Would be good to get some clarification on what we can and can't do. > Cheers, > > Colin > > On Apr 10, 4:18 pm, "Nick Johnson (Google)" <[email protected]> > wrote: > > > > > > > Hi Peter, > > > The simultaneous dynamic request limit was eliminated in a recent SDK. It > > has been replaced with the flexible provisioning system detailed in the FAQ > > entry you quote. > > > It's correct that as long as you keep your latencies low, you should never > > run into any sort of request limit. > > > -Nick Johnson > > > On Sat, Apr 10, 2010 at 2:12 PM, peterk <[email protected]> wrote: > > > I've been using and watching appengine since, well, the beginning, and > > > only by a random Google about an unrelated topic did I find out about > > > this notion of a cap on 'simultaneous active requests'. I came across > > > discussions where people were saying that there is a cap of 30 > > > simultaneous active requests, citing the Quota page and so on. > > > > But when I go to the Quota page - there's not a single mention of any > > > such limit. Why not? Has something changed? > > > > Reading the quotas on that page it's easy to conclude - naively > > > perhaps in hindsight - that 500 requests/second is 500 requests/second > > > irrespective of the average execution time of those requests. > > > > There is no mention of any assumption on that page about the time > > > taken to serve a request or anything else that goes into the > > > calculation for 500 requests/second. > > > > After some further searching I finally found mention of this limit in > > > the FAQs, but here it's still quite vague - and again, no specific > > > mention of this 30 simultaneous request limit. It says: > > > > "However, App Engine reserves automatic scaling capacity for > > > applications with low latency, where the application responds to > > > requests in less than one second. Applications with very high latency > > > (over one second per request for many requests) are limited by the > > > system, and require a special exemption in order to have a large > > > number of simultaneous dynamic requests. If your application has a > > > strong need for a high throughput of long-running requests, you can > > > request an exemption from the simultaneous dynamic request limit. The > > > vast majority of applications do not require any exemption." > > > > Which doesn't imply a hard limit, but rather that as long as your > > > (average?) request fulfilment time is below one second you shouldn't > > > have to worry about this. To be very exacting, if I keep my average > > > request time under 1 second I should *never* see an error relating to > > > this cap? This is a bit different than the talk of 30 simultaneous > > > requests max - at that level with the max of 500 requests a second, > > > you'd be limited to an average request time of 60ms! > > > > So what is the story here? Why is this all being talked about in the > > > shadows of message boards with no clear policy? Why isn't this > > > limitation in the quota page? I thought I had GAE all figured out but > > > then this ball gets thrown into the mix out of left field. > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine%2Bunsubscrib > > > [email protected]> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: > > Registered in Dublin, Ireland, Registration Number: 368047 > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > > 368047 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
