Hi Takashi, Most of my traffic is from other systems, with predictable ramp ups/ down at peak periods during week days (app is cliniconexapp-hrd). Not sure if it would be considered spiky.
Right now I have 1 resident and 4 dynamic. That's more instances than usual though - usually a couple only, and App Engine just seems to be keeping a few extra around right now for whatever reason. Java, multi-threaded. Max idle instances is automatic. The resident instance handles almost no requests (only 3 so far). If a dynamic instance needs to be started, couldn't the scheduler send at least some requests to the idle instance while calling /_ah/warmup on the spawning dynamic one? The main problem with loading requests for me is for an incoming SMS handler for Twilio. Twilio callbacks have a hard timeout of 15 seconds, so if it hits a loading request the user will see a fallback error SMS response. Not ideal, and i'd like to minimize the chances of it. It sounds like I could pay to reserve even more excess capacity to do this, but I'd think that my 1 resident instance could at least be used a bit (say 30% load temporarily) to allow some predictive loading of dynamic instances via /_ah/warmup. /Tom On Jul 14, 9:49 am, Takashi Matsuo <[email protected]> wrote: > On Jul 14, 2012 12:42 PM, "Tom Phillips" <[email protected]> wrote: > > > > > Hi Takashi, > > > Now that I've added a resident idle instance, the logs seem to confirm > > that only the loading of the resident instance is via /_ah/warmup. > > It's difficult to tell that for sure, but all my current dynamic > > instances were loaded on a user URL, and the current resident > > instance was loaded via /_ah/warmup. > > Yes, it is expected behavior. > > > > > Resident idle instances are loaded only infrequently, and serve little > > traffic, so of what use are warmup requests if only they get them? The > > vast majority of loading requests are now on user URLs. That's a 20+ > > second wait (Java) by a user on almost every loading request. > > Those resident instances are used when there is no available dynamic > instance. If there's no available resident instance, your user will see the > loading requests. > > In general, if you see too many user-facing loading requests, there is > still a room for raising your number of resident instances in order to > reduce the number of user-facing loading requests. > > > > > Back in the Always-on days, I would see virtually all loading requests > > use /_ah/warmup, as long as traffic was relatively stable. So an > > instance loading rarely affected an actual user. What do we configure > > now to get this same behavior? > > Probably you can not expect the very same behavior. > > How many resident instances are you configured? Are you configured the 'Max > Idle instances' to 'Automatic'? How spiky is your access pattern? Are you > using the concurrent requests? > > -- Takashi > > > > > > > > > > > > > /Tom > > > On Jul 13, 7:07 pm, Takashi Matsuo <[email protected]> wrote: > > > Hi Mauricio, > > > > Yes I understand that, and we've been working on it. Sorry that it's > taking > > > long. > > > > Hi Wilson, > > > > It's coming with the pricing change. Since we started charging with > number > > > of instances, we needed to provide more control on how we spin-up a new > > > instances. > > > In other words, if we continued the old behavior, many developers would > > > have complained like "App Engine spins up unnecessary instances.". > > > > We believe that it makes more sense when warmup requests are used only > for > > > resident idle instances. > > > > -- Takashi > > > > On Sat, Jul 14, 2012 at 7:18 AM, Wilson MacGyver <[email protected] > >wrote: > > > > > any reason behind this change? this seems kind of strange. I too was > > > > wondering by the > > > > behavior till you posted this. I had my min set to automatic also > > > > > On Fri, Jul 13, 2012 at 9:28 AM, Takashi Matsuo <[email protected]> > > > > wrote: > > > > > > Now the warmup requests are fired only if you set min idle > instances to > > > > some > > > > > value(not automatic). > > > > > > -- Takashi > > > > > > On Fri, Jul 13, 2012 at 9:52 PM, Michael Hermus < > > > > [email protected]> > > > > > wrote: > > > > > >> Same for me.. I just checked: no calls to warmup, lots of loading > > > > >> requests. > > > > > >> **shakes fist at App Engine** > > > > > >> On Thursday, July 12, 2012 4:39:48 PM UTC-4, Tom Phillips wrote: > > > > > >>> Interesting..I checked and I too have 100% of my loading requests > on > > > > >>> user facing URLs instead of /_ah/warmup. > > > > > >>> Warmup requests are enabled and Automatic-Automatic for both > instance > > > > >>> sliders. > > > > > >>> I used to see at least a decent percentage of loading requests on > /_ah/ > > > > >>> warmup, but haven't looked in quite a while. > > > > > >>> /Tom > > > > > >>> On Jul 12, 3:46 pm, David Hardwick <[email protected] > > > > > >>> wrote: > > > > >>> > Some additional observations and questions... > > > > > >>> > After reading this [Link 1] stack overflow article that > mentioned an > > > > >>> > issue with having your Max Idle count below 6, we started > looking at > > > > >>> > our warmup request on our staging environment because that > app-id has > > > > >>> > Idle Instances set to Auto-Auto, while production had specific > > > > values. > > > > > >>> > But...Where did all the "/_ah/warmup" requests go? When doing a > > > > label > > > > >>> > search for these staging environment logs ["path:/_ah/warmup" > (doing > > > > a > > > > >>> > label search)] we couldn't find any warmup request!!(yes, we > have > > > > >>> > warmup requests turned on)...we would just see the first > cold-start > > > > >>> > request would take around 15 seconds to load (F1) and 10 > seconds to > > > > >>> > load on (F2). > > > > > >>> > I even shut down every instance and hit the staging server > again to > > > > >>> > see if I could find a warmup request in the logs...nope. > Honestly, I > > > > >>> > would rather have a user wait 10 seconds for the first request > to > > > > that > > > > >>> > server as opposed risking the warmup requests failing again. > > > > > >>> > Where did all the "/_ah/warmup" requests go? More > importantly, why > > > > >>> > would we have such different times for warmup requests compared > to > > > > >>> > cold starts? Shouldn't they be nearly identical?! > > > > > >>> > Rock on, > > > > >>> > -Hardwick > > > > > >>> > [Link 1] > > > > >>> > - > > > >http://stackoverflow.com/questions/9422698/ah-warmup-producing-hardde. > .. > > > > > >>> > On Jul 12, 12:26 pm, David Hardwick < > > [email protected]> > > > > > > > > > > > >>> > wrote: > > > > > >>> > > Hello, > > > > > >>> > > I realize there's been a lot of discussion on startup times > > > > exceeded > > > > >>> > > on > > > > >>> > > this forum recently, but wanted needed to post this > experience we > > > > had > > > > >>> > > this > > > > >>> > > morning to keep the attention on this important issue. > > > > > >>> > > We uploaded a point release of our app to a "not-live" > version this > > > > >>> > > morning > > > > >>> > > and, of course, we were going to click around on that > instance to > > > > >>> > > make sure > > > > >>> > > it's all kosher before making that version "live." The > warm-up > > > > >>> > > requests > > > > >>> > > for the "not-live" version were exceeding the deadline limit > of > > > > >>> > > 60s... > > > > >>> > > __and__we__are__on__F4s__!_!. > > > > > >>> > > However, the LIVE version of the app crashed too, 500 server > > > > errors, > > > > >>> > > instance counts went to zero, all sorts of whacky stuff was > seen in > > > > >>> > > the > > > > >>> > > control panel. All that happened to our LIVE version without > when > > > > >>> > > all we > > > > >>> > > did was upload another "non-live" version and hit it with a > single > > > > >>> > > request...did I mention we were on F4s? ;-) Does the > failure of > > > > any > > > > >>> > > instance to exceed the 60s limit take down all instances to > include > > > > >>> > > live > > > > >>> > > one? > > > > > >>> > > We did a few things as quickly as possible since our live > > > > application > > > > >>> > > was > > > > >>> > > down, so clearly we didn't have the time to take the > scientific > > > > >>> > > approach of > > > > >>> > > only changing one thing at a time and wait to see if it that > did > > > > it. > > > > > >>> > > We... > > > > >>> > > 1. Switched from F4s to F2 (i figured if this would least get > us on > > > > >>> > > some > > > > >>> > > new servers/instances) > > > > >>> > > 2. Increased max idle instances from 1 to 2 (with F4s > running, I'm > > > > >>> > > fine > > > > >>> > > with having just 1 idle instance and not at all happy about > paying > > > > >>> > > for 2 > > > > >>> > > idle instances, so maybe we'll just increase this prior to > > > > >>> > > deployments and > > > > >>> > > then back down again after the deployment succeeds until we > know > > > > >>> > > more) > > > > >>> > > 3. Made the recently uploaded version live (hey, why not, the > > > > >>> > > production > > > > >>> > > app was down for 10 minutes, so how much more harm could we > do?) > > > > > >>> > > We use GWT and Guice, we jar everything (as I have been paying > > > > >>> > > attention to > > > > >>> > > this startup time discussions for quite some time now. We > are also > > > > >>> > > considering switching our Guice libraries to a non-AOP > version as > > > > we > > > > >>> > > saw > > > > >>> > > suggested in another blog since we just need the injection. > > > > > >>> > > Any insight, and I'm all ears! app_id=s~myflashpanel > > > > > >>> > > Regards, > > > > >>> > > -Hardwick > > > > > >>> > > -- > > > > > >>> > > *We make Google Apps even better.* > > > > > >>> > > *David Hardwick* > > > > >>> > > *CTO* > > > > >>> > > [email protected] > > > > > >>> > > *Signature by Flashpanel <http://flashpanel.com/>* > > > > >>> > > *See us in Mashable: Growing Up Google: How Cloud Computing > Is > > > > >>> > > Changing a > > > > >>> > > Generation > > > > >>> > > <http://mashable.com/2012/04/30/generation-growing-up-google/ > >* > > > > > >> -- > > > > >> You received this message because you are subscribed to the Google > > > > Groups > > > > >> "Google App Engine" group. > > > > >> To view this discussion on the web visit > > > > >>https://groups.google.com/d/msg/google-appengine/-/Bs6JKwLYDAMJ. > > > > > >> To post to this group, send email to > > [email protected].> > > >> To unsubscribe from this group, > send email to > > > > >> [email protected]. > > > > >> For more options, visit this group at > > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > > Takashi Matsuo > > > > > > -- > > > > > You received this message because you are subscribed to the Google > Groups > > > > > "Google App Engine" group. > > > > > To post to this group, send email to > > [email protected]. > > > > > > > > > > > > To unsubscribe from this group, send email to > > > > > [email protected]. > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > > Omnem crede diem tibi diluxisse > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
