Also FYI, this whole issue could be resolved if you would stop sending user facing requests to cold uninitialized instances in java GAE. Handling user requests in ways that you know will cause 20+ second response times is pathological.
All new development for us on GAE is blocked until this issue can be resolved. On Thursday, December 27, 2012 11:01:30 AM UTC-8, Carl Schroeder wrote: > > I am still seeing java instances decomissioned after sub minute quiet > periods. Given that it takes 20-30 seconds to spin one java instance up, > you should probably leave them alive for a bit longer than a few seconds. > Otherwise, for low traffic profiles, page loads for GAE java can take up to > 30 seconds. God help us if the scheduler thinks I need 2 new instances spun > up in series rather than parallel. > > FYI, people don't wait around for a minute for pages to load. They use > other services. > > Once again, due to unannounced pathological behavior of the instancing on > GAE, we are wasting our time re-implementing our java infrastructure on > AWS. At least, I hope it is a waste of our time... > > > On Tuesday, December 25, 2012 4:00:26 PM UTC-8, Francois Masurel wrote: >> >> Hi Saket, >> >> Thanx for investigating the problem. >> >> Things seems to have improved significantly these last few days. >> >> On our Neustar reports, for an average page loading, we can see that we >> went from >> 6.65s<https://monitor.wpm.neustar.biz/reports/graph_log/#/?monitorIds=8bc3b100-b372-11e1-a749-9848e1660ab3&timeRange=last_24_hours&functionType=loadtime&graphType=line_chart&view=loadtime> >> (12/16) >> to 8.94s (12/18) and back to >> 2.57s<https://monitor.wpm.neustar.biz/reports/graph_log/#/?monitorIds=8bc3b100-b372-11e1-a749-9848e1660ab3&timeRange=last_24_hours&functionType=loadtime&graphType=line_chart&view=loadtime> >> (12/23). >> >> I can confirm that we don't see as much instance warm ups in our logs as >> we were used before, we went from every 3-4 minutes to 20-25 minutes. >> >> I definitely think things have changed on Google side as we haven't >> changed anything on our side. >> >> But some strange things are still going on at the moment (12/26 0:52 >> UTC+1) as a dynamic instance has been up for more than 4 hours but has >> served only one request (cf. screenshot below). >> >> Thanx again Saket for your help. >> >> François >> >> >> <https://lh6.googleusercontent.com/-ARDAA2Yu-NY/UNo-Bgn9SpI/AAAAAAAAyN8/NNC4QLzi93g/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg> >> >> >> >> >> On Monday, December 24, 2012 11:54:48 PM UTC+1, Saket Kumar wrote: >>> >>> Hi Francois, >>> >>> Are you still facing this issue? I did a small test for your application >>> and didn't find anything too bad from scheduler's perspective. I'm trying >>> to understand if it was a temporary glitch that was causing the issue or >>> something is wrong with scheduler's algorithm. Or if scheduler doesn't spin >>> instances properly if QPS is low. >>> >>> Here is series of events- >>> >>> a.) Single resident was serving for 1 day, 03:26:37 minutes, serving >>> 9600 request >>> b.) After serving 96 requests with raised QPS, new instance was created. >>> c.) QPS lowered and the new instance was allowed to die. >>> d.) Again the QPS was increased and new instance was created. Both >>> instances were handling requests at this point. >>> e.) QPS lowered again, newly formed instance dies and older instance >>> starts serving 100% of the requests. >>> >>> - >>> Saket >>> >>> On Saturday, 22 December 2012 12:18:00 UTC+1, Francois Masurel wrote: >>>> >>>> Hi Christina, >>>> >>>> For my personal case, my production app-id is "vncts1" with billing >>>> enabled. >>>> >>>> And the related issue is : >>>> https://code.google.com/p/googleappengine/issues/detail?id=7865 >>>> >>>> As I said at the beginning of this thread, there are quite a few other >>>> threads related to this problem : >>>> >>>> >>>> https://groups.google.com/forum/#!msg/google-appengine/cJcOrOE4JDc/uKFBbylFOxIJ >>>> >>>> https://groups.google.com/d/msg/google-appengine/8KiXRP-oU5I/Jf5kzcvyIiwJ >>>> >>>> https://groups.google.com/d/msg/google-appengine/NR_oV4znvks/KtkSmVQD2IgJ >>>> >>>> https://groups.google.com/d/msg/google-appengine/_znmrJwFMuM/V2epYOQ19L4J >>>> >>>> https://groups.google.com/d/topic/google-appengine/hg3BH8WOGWA/discussion >>>> >>>> https://groups.google.com/d/topic/google-appengine/UN5oCWkPsaA/discussion >>>> >>>> https://groups.google.com/d/topic/google-appengine/RneC9qDbeRE/discussion >>>> >>>> https://groups.google.com/d/topic/google-appengine/sLZcZoK-A3Y/discussion >>>> >>>> Thanx for you help. >>>> >>>> François >>>> >>>> >>>> >>>> On Saturday, December 22, 2012 3:15:05 AM UTC+1, Christina Ilvento >>>> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> Would you mind sending app-ids that you're seeing this behavior for? >>>>> Please feel free to send them to me directly or to link any issues you >>>>> have >>>>> filed in our issue tracker so that we can investigate. >>>>> >>>>> >>>>> Thanks, >>>>> Christina >>>>> >>>>> >>>>> On Fri, Dec 21, 2012 at 8:39 AM, Carl Schroeder <[email protected] >>>>> > wrote: >>>>> >>>>>> I've been sacrificing unicorns to dark powers...clearly I have been >>>>>> doing it wrong. >>>>>> That probably explains some unusual behavior in other parts of my app. >>>>>> AFK pentagrams. :( >>>>>> >>>>>> >>>>>> On Thursday, December 20, 2012 7:51:35 PM UTC-8, Cesium wrote: >>>>>>> >>>>>>> Ready for this? >>>>>>> >>>>>>> Now, a single instance survives for hours and hours, happily serving >>>>>>> requests with the usual low latency response time. >>>>>>> >>>>>>> This is just what Sir Brandon wrote about. Mysterious changes in the >>>>>>> system's behavior. >>>>>>> >>>>>>> I should note that I sprinkled rainbow Skittles across the floor to >>>>>>> attract the unicorns. They're back! >>>>>>> >>>>>>> David >>>>>>> >>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msg/google-appengine/-/MFdFH0ZqWbgJ. >>>>>> >>>>>> To post to this group, send email to [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Christina Ilvento | Google App Engine | [email protected] | >>>>> (650)-201-9399 >>>>> >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/IB1o3Nu1vmkJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
