I should have looked at this thread before posting my own (http:// groups.google.com/group/google-appengine/browse_thread/thread/ 7ff808e8243562cf#).
I'm using GAE as a sync-backend for a smallish app, so my traffic is very bursty, with long idle periods. I've been seeing exactly what you describe for the past few days, although I haven't changed my scheduler settings in many months (1 idle instance, 7s min pending latency.) Maybe they changed something so that the scheduler is now avoiding sending requests to that one idle instance at all costs. This results in requests sitting in the pending queue for those 7 seconds, and then a new instance is started - so many requests to my idle app are waiting for 10s now, while the idle instance does exactly nothing. The fact that this behavior has been coming and going, and also that normal (non cold-start) requests have lately been much slower than normal, leads me to suspect that this isn't a deliberate change but some kind of instability in GAE that's been going on for days now. I'd love to hear some feedback as to what's happening. On Feb 29, 2:02 pm, Simon Knott <[email protected]> wrote: > Hi, > > I thought I was going crazy - in the past week or so the latter is exactly > what I'm experiencing. My free-tier app gets so little traffic at the > moment that I was putting it down to that, but maybe it's not. > > I've tested it with just a single user and I'm seeing three instances > started for a single page request, which has a three ajax calls on it. > What I see is the following: > > - Static page loads which fires 3 ajax calls > - New instance gets started for the first ajax call > - Second ajax call also starts up a new instance, but the initialisation > is delayed until the first ajax call completes > - Third ajax call also starts up a new instance, but the initialisation > is delayed until the second ajax call completes > > All in all, the single page is taking 20+ seconds to load and ends up with > three instances. I don't have a lot of third-party libraries and > initialisation of the server used to take 2-3 seconds, which seems to have > elevated to 7 or 8 seconds in the last couple of weeks. Even once the page > is loaded, another ajax call can then trigger a new instance to be started > - it's bonkers. There doesn't appear to be any concurrency, or reuse of > existing instances, at all. It is currently marked as thread-safe=true. > > Cheers, > Simon > > > > > > > > > > On Tuesday, 28 February 2012 17:33:14 UTC, Jeff Schnitzer wrote: > > > There's been a lot of discussion of the scheduler behavior in Pythonland, > > but not much about it's "eccentricities" in Javaland. > > > I have a threadsafe=true Java app. Let's say every request completes in > > exactly 1s. Settings are: idle instances min 1 max 1, latency auto/auto. > > Here is what I expect: > > > * Instance1 starts up and becomes permanently resident > > * Instance1 serves concurrent requests up to some arbitrary CPU capcity > > * When Instance1 exceeds capacity: > > * Instance2 starts warming up > > * All requests remain in the pending queue for Instance1, getting > > processed at 1/s * concurrency > > * Instance2 is ready and starts processing new requests, sharing the > > load with Instance1 > > > What I actually see (as far as I can determine): > > > * Instance1 starts up and becomes permanently resident > > * Instance1 supports almost no concurrency. At most it's 2. (no, my app > > is not particularly compute intensive) > > * A new request comes in which for some reason can't be handled by > > Instance1: > > * Instance2 starts warming up > > * The new request is blocked on Instance2's pending queue, waiting > > 10-20s for Instance2 to be ready > > * In the mean time, Instance1 is actually idle > > * Another new request comes in and starts up Instance3 > > * Possibly this is while Instance2 is warming up > > * AFAICT, Instance1 is taking a coffee break > > > The net result is that I have an idle website with 1 user (me) clicking > > around and I've already gotten multiple 20s pauses and three instances. > > Something is seriously wrong here. Whether or not it's rational to have > > so many instances started, pending requests shouldn't be shunted to > > non-warmed-up servers, right? > > > I've tried upping the min latency to a high value to see if this improves > > the situation. If this works... shouldn't min latency *always* be as high > > as the startup time for an instance? > > > I know it's been said before, but it needs to be said again... the > > guidance for scheduler configuration is really, really inadequate. > > > Jeff > > On Tuesday, 28 February 2012 17:33:14 UTC, Jeff Schnitzer wrote: > > > There's been a lot of discussion of the scheduler behavior in Pythonland, > > but not much about it's "eccentricities" in Javaland. > > > I have a threadsafe=true Java app. Let's say every request completes in > > exactly 1s. Settings are: idle instances min 1 max 1, latency auto/auto. > > Here is what I expect: > > > * Instance1 starts up and becomes permanently resident > > * Instance1 serves concurrent requests up to some arbitrary CPU capcity > > * When Instance1 exceeds capacity: > > * Instance2 starts warming up > > * All requests remain in the pending queue for Instance1, getting > > processed at 1/s * concurrency > > * Instance2 is ready and starts processing new requests, sharing the > > load with Instance1 > > > What I actually see (as far as I can determine): > > > * Instance1 starts up and becomes permanently resident > > * Instance1 supports almost no concurrency. At most it's 2. (no, my app > > is not particularly compute intensive) > > * A new request comes in which for some reason can't be handled by > > Instance1: > > * Instance2 starts warming up > > * The new request is blocked on Instance2's pending queue, waiting > > 10-20s for Instance2 to be ready > > * In the mean time, Instance1 is actually idle > > * Another new request comes in and starts up Instance3 > > * Possibly this is while Instance2 is warming up > > * AFAICT, Instance1 is taking a coffee break > > > The net result is that I have an idle website with 1 user (me) clicking > > around and I've already gotten multiple 20s pauses and three instances. > > Something is seriously wrong here. Whether or not it's rational to have > > so many instances started, pending requests shouldn't be shunted to > > non-warmed-up servers, right? > > > I've tried upping the min latency to a high value to see if this improves > > the situation. If this works... shouldn't min latency *always* be as high > > as the startup time for an instance? > > > I know it's been said before, but it needs to be said again... the > > guidance for scheduler configuration is really, really inadequate. > > > Jeff > > On Tuesday, 28 February 2012 17:33:14 UTC, Jeff Schnitzer wrote: > > > There's been a lot of discussion of the scheduler behavior in Pythonland, > > but not much about it's "eccentricities" in Javaland. > > > I have a threadsafe=true Java app. Let's say every request completes in > > exactly 1s. Settings are: idle instances min 1 max 1, latency auto/auto. > > Here is what I expect: > > > * Instance1 starts up and becomes permanently resident > > * Instance1 serves concurrent requests up to some arbitrary CPU capcity > > * When Instance1 exceeds capacity: > > * Instance2 starts warming up > > * All requests remain in the pending queue for Instance1, getting > > processed at 1/s * concurrency > > * Instance2 is ready and starts processing new requests, sharing the > > load with Instance1 > > > What I actually see (as far as I can determine): > > > * Instance1 starts up and becomes permanently resident > > * Instance1 supports almost no concurrency. At most it's 2. (no, my app > > is not particularly compute intensive) > > * A new request comes in which for some reason can't be handled by > > Instance1: > > * Instance2 starts warming up > > * The new request is blocked on Instance2's pending queue, waiting > > 10-20s for Instance2 to be ready > > * In the mean time, Instance1 is actually idle > > * Another new request comes in and starts up Instance3 > > * Possibly this is while Instance2 is warming up > > * AFAICT, Instance1 is taking a coffee break > > > The net result is that I have an idle website with 1 user (me) clicking > > around and I've already gotten multiple 20s pauses and three instances. > > Something is seriously wrong here. Whether or not it's rational to have > > so many instances started, pending requests shouldn't be shunted to > > non-warmed-up servers, right? > > > I've tried upping the min latency to a high value to see if this improves > > the situation. If this works... shouldn't min latency *always* be as high > > as the startup time for an instance? > > > I know it's been said before, but it needs to be said again... the > > guidance for scheduler configuration is really, really inadequate. > > > Jeff > > On Tuesday, 28 February 2012 17:33:14 UTC, Jeff Schnitzer wrote: > > > There's been a lot of discussion of the scheduler behavior in Pythonland, > > but not much about it's "eccentricities" in Javaland. > > > I have a threadsafe=true Java app. Let's say every request completes in > > exactly 1s. Settings are: idle instances min 1 max 1, latency auto/auto. > > Here is what I expect: > > > * Instance1 starts up and becomes permanently resident > > * Instance1 serves concurrent requests up to some arbitrary CPU capcity > > * When Instance1 exceeds capacity: > > * Instance2 starts warming up > > * All requests remain in the pending queue for Instance1, getting > > processed at 1/s * concurrency > > * Instance2 is ready and starts processing new requests, sharing the > > load with Instance1 > > > What I actually see (as far as I can determine): > > > * Instance1 starts up and becomes permanently resident > > * Instance1 supports almost no concurrency. At most it's 2. (no, my app > > is not particularly compute intensive) > > * A new request comes in which for some reason can't be handled by > > Instance1: > > * Instance2 starts warming up > > * The new request is blocked on Instance2's pending queue, waiting > > 10-20s for Instance2 to be ready > > * In the mean time, Instance1 is actually idle > > * Another new request comes in and starts up Instance3 > > * Possibly this is while Instance2 is warming up > > * AFAICT, Instance1 is taking a coffee break > > > The net result is that I have an idle website with 1 user (me) clicking > > around and I've already gotten multiple 20s pauses and three instances. > > Something is seriously wrong here. Whether or not it's rational to have > > so many instances started, pending requests shouldn't be shunted to > > non-warmed-up servers, right? > > > I've tried upping the min latency to a high value to see if this improves > > the situation. If this works... shouldn't min latency *always* be as high > > as the startup time for an instance? > > > I know it's been said before, but it needs to be said again... the > > guidance for scheduler configuration is really, really inadequate. > > > Jeff > > On Tuesday, 28 February 2012 17:33:14 UTC, Jeff Schnitzer... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
