You are right. That's precisely why I said "somehow" there--scheduler can go wrong. Maybe the app had a little spike for a sec or it's really just tablet your instance were running on had a little hiccup. Actually, it seems like old instance were plain dead to scheduler for a while.
Here it starts to be interesting, even though we had max latency = 15s and new instances with 20s latency were ineligible, it only started like three instances until the scheduler figured out that it doesn't help and decided to fed requests to existing ones. Then performance settings come into the game: as scheduler "somehow" found instance to be slow it was trying to use new, "hotter" instances (screenshots 3-4) and nicely sent only two request to the old one. And since we haven't explicitly stated how much idle instances we want, the scheduler might have guessed (albeit, mistakenly) -- OK, this app needs two instances right now. Just my speculative optimistic assumptions, but it's kind of remarkable. It'd be cool if the scheduler did improve that much. I've had worst: absolutely no traffic other than me, yet new instances were spinning off with no apparent reason. The issue have disappeared within a couple days. I am to wordy again, so: try to set max latency to auto, plus add a resident instance for a while. Even if problem stays, it should keep scheduler from running new instances. On Sun, Nov 18, 2012 at 2:49 AM, Francois Masurel <[email protected]>wrote: > Hi Igor, > > Thanx for your nice and long answer. I understand and I'm ok with most of > what you said. > > The only thing that annoys me is that my app is a Java app with threading > enabled and as you can see on the screenshots the main long-lived active > instance doesn't do anything : extremely low latency and almost no traffic > served (no traffic at all between second and third screenshot). > > So, no, the active instance's queue is not full unless everything goes > nowhere or the dashboard requests count is wrong. > > What do you think ? > > Thanx again for your help. > > François > > PS: > > And without changing any settings everything has been working fine for > months. > > > On Saturday, November 17, 2012 6:23:36 PM UTC+1, Igor Kharin wrote: > >> I've seen such pattern with my apps. As far as I can see, I believe, this >> is perfectly OK given your performance settings. >> >> Somehow your active instance's "queue" was full and scheduler decided to >> start a new instance. Since it was trying to serve the request for 20 >> seconds, and again, given your settings (max latency = 15s), scheduler >> found it bad and kept trying to send requests to the old instance. And >> again. And again. >> >> There are like exactly two safe options for performance settings: >> 1. min idle instances = auto, max = 1, and a fairly high min latency (low >> cost); >> 2. min idle instances = 1+, max = auto (traffic spike); >> >> Generally, AE engineers recommend not to mix "min" and "max" options and >> always set the opposite slider to "auto", as settings like in your case >> (min latency = 10s, max = 15s) confuse scheduler's heuristics. My advice >> would be to set "max pending latency" to auto and see what happens. Then >> tune the "idle instances" setting as desired: >> 1. max idle instances = 1 so you wouldn't pay for surplus instances; >> 2. OR set min idle instances to 1+ as you obviously seeing a little spike. >> >> Also, keep in mind that resident instances are not really what we had >> with Always On--treat them as "fallback instances". To wrap up, in your >> case, if you want to keep users off of startup requests completely (20s is >> no good): >> 1. set up a cron "ping" task to keep your dynamic instance "warm"; >> 2. set min idle instances to 1 (do not forget to handle warmup requests); >> 3. tune latency settings (say, min = auto, max = 1s to try to evenly >> distribute load between resident and dynamic instances); >> >> Hope that'll help. >> >> >> On Sat, Nov 17, 2012 at 10:44 PM, Francois Masurel <[email protected]>wrote: >> >>> It looks like old instances cannot serve any new coming requests : >>> >>> <https://lh4.googleusercontent.com/-aWdte4xv3WM/UKeuUzRBLjI/AAAAAAAAwOM/90Ii-6MyXbI/s1600/Instances+-+VinoCities+-+Google+Chrome_2.jpg> >>> >>> >>> <https://lh5.googleusercontent.com/-XvW9NIfFJr4/UKeu5W-8-PI/AAAAAAAAwOU/RANrlkjySt4/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg> >>> >>> >>> <https://lh4.googleusercontent.com/-LnT3CjB47Cg/UKevT68Rt8I/AAAAAAAAwOc/ITOjbF_wzF0/s1600/Instances+-+VinoCities+-+Google+Chrome_3.jpg> >>> >>> >>> <https://lh5.googleusercontent.com/-U6MzMOBSMho/UKewNjofMkI/AAAAAAAAwOs/Hc4nw1NW8ro/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg> >>> >>> Looks like GAE send almost all traffic to the new instance which is not >>> fully initialized (lazy loading). >>> >>> App Id : vncts1 >>> >>> On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote: >>> >>>> I've notice lots of user facing loading requests since a few hours >>>> (days?). >>>> >>>> Just check the screenshot below. Two new instances have been started >>>> to serve only 13 (12 + 1) requests though another instance was already >>>> running and available and obviously was not overloaded as it had only >>>> served 105 requests during the last hour : >>>> >>>> >>>> <https://lh4.googleusercontent.com/-UxI0M8fnvw0/UKep9Wkb_VI/AAAAAAAAwN0/wxKaPhZvXlI/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg> >>>> >>>> My application instance settings are all set to automatic and pending >>>> latency set to 10s : >>>> >>>> >>>> <https://lh4.googleusercontent.com/-TxgVTyTvqJM/UKeqaNNQvBI/AAAAAAAAwN8/fKUP3ftluk8/s1600/Application+Settings+-+VinoCities+-+Google+Chrome.jpg> >>>> >>>> Did something change on GAE side recently ? >>>> >>>> These user facing loading requests are really killing my business as my >>>> users are getting really upset about it. >>>> >>>> Thanx for your help. >>>> >>>> François >>>> >>>> App Id = vncts1 >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit https://groups.google.com/d/** >>> msg/google-appengine/-/**ylBW4ldXw_cJ<https://groups.google.com/d/msg/google-appengine/-/ylBW4ldXw_cJ> >>> . >>> >>> To post to this group, send email to google-a...@googlegroups.**com. >>> To unsubscribe from this group, send email to google-appengi...@** >>> googlegroups.com. >>> >>> For more options, visit this group at http://groups.google.com/** >>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>> . >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/uErXDY8EB9UJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
