The related issue is getting some activity lately: http://code.google.com/p/googleappengine/issues/detail?id=7865
On Sunday, November 18, 2012 2:06:35 AM UTC+1, Francois Masurel wrote: > > Hi Igor, > > I have followed your advice and set max latency to auto and min idle > instance to 1 : > > > <https://lh6.googleusercontent.com/-XTTg4ebm-mY/UKg0XDUdozI/AAAAAAAAwO8/V--InrAlRGc/s1600/Application+Settings+-+VinoCities+-+Google+Chrome.jpg> > > I'll tell you how it goes. > > Thanx again for your help. > > François > > > On Saturday, November 17, 2012 10:50:07 PM UTC+1, Igor Kharin wrote: >> >> You are right. That's precisely why I said "somehow" there--scheduler can >> go wrong. Maybe the app had a little spike for a sec or it's really just >> tablet your instance were running on had a little hiccup. Actually, it >> seems like old instance were plain dead to scheduler for a while. >> >> Here it starts to be interesting, even though we had max latency = 15s >> and new instances with 20s latency were ineligible, it only started like >> three instances until the scheduler figured out that it doesn't help and >> decided to fed requests to existing ones. Then performance settings come >> into the game: as scheduler "somehow" found instance to be slow it was >> trying to use new, "hotter" instances (screenshots 3-4) and nicely sent >> only two request to the old one. And since we haven't explicitly stated how >> much idle instances we want, the scheduler might have guessed (albeit, >> mistakenly) -- OK, this app needs two instances right now. >> >> Just my speculative optimistic assumptions, but it's kind of remarkable. >> It'd be cool if the scheduler did improve that much. I've had worst: >> absolutely no traffic other than me, yet new instances were spinning off >> with no apparent reason. The issue have disappeared within a couple days. I >> am to wordy again, so: try to set max latency to auto, plus add a resident >> instance for a while. Even if problem stays, it should keep scheduler from >> running new instances. >> >> >> On Sun, Nov 18, 2012 at 2:49 AM, Francois Masurel <[email protected]>wrote: >> >>> Hi Igor, >>> >>> Thanx for your nice and long answer. I understand and I'm ok with most >>> of what you said. >>> >>> The only thing that annoys me is that my app is a Java app with >>> threading enabled and as you can see on the screenshots the main long-lived >>> active instance doesn't do anything : extremely low latency and almost no >>> traffic served (no traffic at all between second and third screenshot). >>> >>> So, no, the active instance's queue is not full unless everything goes >>> nowhere or the dashboard requests count is wrong. >>> >>> What do you think ? >>> >>> Thanx again for your help. >>> >>> François >>> >>> PS: >>> >>> And without changing any settings everything has been working fine for >>> months. >>> >>> >>> On Saturday, November 17, 2012 6:23:36 PM UTC+1, Igor Kharin wrote: >>> >>>> I've seen such pattern with my apps. As far as I can see, I believe, >>>> this is perfectly OK given your performance settings. >>>> >>>> Somehow your active instance's "queue" was full and scheduler decided >>>> to start a new instance. Since it was trying to serve the request for 20 >>>> seconds, and again, given your settings (max latency = 15s), scheduler >>>> found it bad and kept trying to send requests to the old instance. And >>>> again. And again. >>>> >>>> There are like exactly two safe options for performance settings: >>>> 1. min idle instances = auto, max = 1, and a fairly high min latency >>>> (low cost); >>>> 2. min idle instances = 1+, max = auto (traffic spike); >>>> >>>> Generally, AE engineers recommend not to mix "min" and "max" options >>>> and always set the opposite slider to "auto", as settings like in your >>>> case >>>> (min latency = 10s, max = 15s) confuse scheduler's heuristics. My >>>> advice would be to set "max pending latency" to auto and see what happens. >>>> Then tune the "idle instances" setting as desired: >>>> 1. max idle instances = 1 so you wouldn't pay for surplus instances; >>>> 2. OR set min idle instances to 1+ as you obviously seeing a little >>>> spike. >>>> >>>> Also, keep in mind that resident instances are not really what we had >>>> with Always On--treat them as "fallback instances". To wrap up, in your >>>> case, if you want to keep users off of startup requests completely (20s is >>>> no good): >>>> 1. set up a cron "ping" task to keep your dynamic instance "warm"; >>>> 2. set min idle instances to 1 (do not forget to handle warmup >>>> requests); >>>> 3. tune latency settings (say, min = auto, max = 1s to try to evenly >>>> distribute load between resident and dynamic instances); >>>> >>>> Hope that'll help. >>>> >>>> >>>> On Sat, Nov 17, 2012 at 10:44 PM, Francois Masurel >>>> <[email protected]>wrote: >>>> >>>>> It looks like old instances cannot serve any new coming requests : >>>>> >>>>> <https://lh4.googleusercontent.com/-aWdte4xv3WM/UKeuUzRBLjI/AAAAAAAAwOM/90Ii-6MyXbI/s1600/Instances+-+VinoCities+-+Google+Chrome_2.jpg> >>>>> >>>>> >>>>> <https://lh5.googleusercontent.com/-XvW9NIfFJr4/UKeu5W-8-PI/AAAAAAAAwOU/RANrlkjySt4/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg> >>>>> >>>>> >>>>> <https://lh4.googleusercontent.com/-LnT3CjB47Cg/UKevT68Rt8I/AAAAAAAAwOc/ITOjbF_wzF0/s1600/Instances+-+VinoCities+-+Google+Chrome_3.jpg> >>>>> >>>>> >>>>> <https://lh5.googleusercontent.com/-U6MzMOBSMho/UKewNjofMkI/AAAAAAAAwOs/Hc4nw1NW8ro/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg> >>>>> >>>>> Looks like GAE send almost all traffic to the new instance which is >>>>> not fully initialized (lazy loading). >>>>> >>>>> App Id : vncts1 >>>>> >>>>> On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel >>>>> wrote: >>>>> >>>>>> I've notice lots of user facing loading requests since a few hours >>>>>> (days?). >>>>>> >>>>>> Just check the screenshot below. Two new instances have been started >>>>>> to serve only 13 (12 + 1) requests though another instance was already >>>>>> running and available and obviously was not overloaded as it had only >>>>>> served 105 requests during the last hour : >>>>>> >>>>>> >>>>>> <https://lh4.googleusercontent.com/-UxI0M8fnvw0/UKep9Wkb_VI/AAAAAAAAwN0/wxKaPhZvXlI/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg> >>>>>> >>>>>> My application instance settings are all set to automatic and pending >>>>>> latency set to 10s : >>>>>> >>>>>> >>>>>> <https://lh4.googleusercontent.com/-TxgVTyTvqJM/UKeqaNNQvBI/AAAAAAAAwN8/fKUP3ftluk8/s1600/Application+Settings+-+VinoCities+-+Google+Chrome.jpg> >>>>>> >>>>>> Did something change on GAE side recently ? >>>>>> >>>>>> These user facing loading requests are really killing my business as >>>>>> my users are getting really upset about it. >>>>>> >>>>>> Thanx for your help. >>>>>> >>>>>> François >>>>>> >>>>>> App Id = vncts1 >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To view this discussion on the web visit https://groups.google.com/d/* >>>>> *msg/google-appengine/-/**ylBW4ldXw_cJ<https://groups.google.com/d/msg/google-appengine/-/ylBW4ldXw_cJ> >>>>> . >>>>> >>>>> To post to this group, send email to google-a...@googlegroups.**com. >>>>> To unsubscribe from this group, send email to google-appengi...@** >>>>> googlegroups.com. >>>>> >>>>> For more options, visit this group at http://groups.google.com/** >>>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>>>> . >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine/-/uErXDY8EB9UJ. >>> >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/kn-99zDl5q8J. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
