@Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing.
For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under "no load" with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only "1" request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine <[email protected]> wrote: > After speaking with Engs, I think I can explain what is going on: > > Here are the current scheduling rules: (> reads as has priority for > handling the incoming request) > > 1/ Idle Always On instance > Spawning a new Dynamic instance > 2/ Spawning a new Dynamic instance > Busy Always On instance > 3/ Idle Dynamic instance > Busy Always On instance > 4/ Idle Dynamic instance > Idle Always On instance > > I will give you an example to illustrate the behavior you all noticed, > that is Dynamic instance handling request while Always On is idle. > > (Always On instance started) > - Incoming request > - Always On instance handle the request > - another Incoming request > (Always On instance busy) > - A new Dynamic instance is spawned > (Dynamic instance idle, Always on instance busy) > - Dynamic instance handle the request > - another Incoming request > (Dynamic instance idle, Always on instance idle) > - Dynamic instance handle the request > - No request for more than idle-dynamic-instance-timeout > - Dynamic instance shut down > - another Incoming request > (Always On instance idle) > - Always On instance handle the request > > Hope it makes thing clearer. > > As part of the new billing model you will have a scheduler knob called > 'max-idle-instances' that you can use if extra idling dynamic > instances are undesired. > > The good news is that we are open to suggestion, if you think this > behavior is the wrong default, feel free to comment on that thread and > I will follow up your suggestion to the Engineering team. > > > > > > > > > > On Wed, Jul 20, 2011 at 12:18 AM, Galoch <[email protected]> wrote: > > Same here. Seems like GAE is totally ignoring Always On instances. > > I also noticed that even with no user hitting our app and a single > > cron job that runs every 5 minutes it is still spinning instances > > every 3 minutes and then killing them in 2 minutes. > > > This has been happening since after the upgrade on 14th July. During > > peak load this really gets nasty and brings down the performance. > > > This is the feedback I got yesterday from one of our customers since > > it takes time to spin an instance (and yes we use Spring): > > > "1) I found the GUI to be very laggy" > > > Can someone from Google please respond? > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > Johan Euphrosine (proppy) > Developer Programs Engineer > Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
