I spoke too soon. Some time after posting this, testing revealed more problems. Upon diagnosis, it appears that GAE is too aggressive at spinning up new instances even when the "auto/auto" setting is used, although the behavior is less reproducible than when a min of 1 is set.
The only fix I found for this was to switch to using a backend. With a backend I can control exactly how many instances are used. A single instance is sufficient for me and the RTT is now down to about 400-500ms (measured from the client), which is completely satisfactory. What is *not* so satisfactory is (a) backends are more expensive than frontends, and I'm basically paying in order not to use a feature, which I find pretty annoying; and (b) this breaks my site in half and I will now have to worry about cross-domain scripting when I eventually get to the point of having an actual web interface (and not just APIs called by a native app). Something is clearly broken about frontend instance balancing. I'm guessing it's only broken at the lower end of the traffic scale or no one would be using GAE, but it's broken nonetheless. You could argue that GAE is built to scale up, not down, but GAE balancing is currently an *order of magnitude* worse than a naive solution for this problem domain; so even though the pervasiveness might be low, the severity is very high. IMHO this needs to be fixed. - Kris On Tuesday, July 31, 2012 11:35:50 PM UTC-7, Kristopher Giesing wrote: > > The above post notwithstanding, the original problem I was having - which > caused me to try to set min instances to 1 to begin with - seems to have > gone away. I know this is probably going to fall into the PEBCAK bucket, > but I really don't think I did anything in particular to cause the problem, > so I will explain in somewhat tedious detail the circumstances surrounding > the problem, even if some of them aren't relevant. > > The original problem was that the client-side latency of requests was > intermittently very high. I don't have many details on this because as > soon as I tried to diagnose it I ran into the problem outlined above, which > certainly confused my investigation. > > However, now that I realize that the min instance slider is basically > useless to me, I went back to testing against the auto/auto settings which > I was using to begin with. I then noticed that the appstats endpoint was > getting hit at times I didn't expect. To try to diagnose the problem with > A/B testing I deleted all references to appstats and re-deployed my app. > Sure enough, the mysteriously high latency disappeared. Confident I was > on to something, I put appstats back, and the problem failed to reappear. > > So now the performance of the app is in the range that I would expect. > The client-side latency is around a second, which is not great by many > standards, but is sort-of OK for my application and is certainly better > than 5+ seconds. It's also closer to what I see inside the request > handling itself - that is, when client latency is higher, so is the > accumulated latency inside the request handler (from internal data > manipulation etc). This gives me the lever of optimizing my own calls to > avoid occasionally-expensive operations, so it's much less of an issue; > prior to this, the vast majority of the latency happened outside the > request handling where I don't have any control. > > If the performance of GAE generally is in the range I'm seeing now, that's > a shippable product in my mind. I am worried, however, that the problem > was somewhere else in the system and somewhere down the line I will be hit > with this problem again, only it will be in production and users will > rightly complain that their experience is suffering. > > So any further comment on this issue would be very welcome. I'm still > considering whether a backend is a more appropriate vehicle for this > functionality. > > Thanks, > > - Kris > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_v9CmfpUmQ4J. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
