Hi Mike, I can help explain some of the issues going on here. For starters, if an app is thread-safe, then yes that is a very important signal to the scheduler. It indicates that an instance can sustain more requests and as such there is no need to place the incoming request on a pending queue (assuming no other idle instances).
The next most important thing I should point out is that the actual formula for billing purposes for frontend instances is not the "total instances" (blue line) on your instances dashboard graph. If you set max-idle-instances (it appears you did this for most of 09-01 and half of 09-02, and then again earlier today), then the actual formula will be based on both total-instances (blue line) and active-instances (orange line): billable instances = min(active-instances + max-idle-instances, total-instances) So, in your example, active-instances seems to be 0.1-0.2, total-instances is between 2 and 3, and max-idle-instances was 1 during certain time windows. This explains why your billed frontend instances hours were 51.69, 54.82, 30.82, and 41.67 respectively on 08-30, 08-31, 09-01, and 09-02. The days where you had max-idle-instances=1 more, the lower the billed instances hours were. This, even though the total number of instances was largely unmodified. This is because max-idle-instances has two primary functions. The first is to advise the scheduler to physically kill off excess idle instances, and the latter is to be interpreted as a hard ceiling (per the above formula) by our billing system. This is why on 09-01, you were billed based on an average rate of 1.28 instance-seconds-per-second (the other noise in this figure is that the max-idle-instances setting was made at 02:56:41, so did not apply for the first ~three hours of the day). Regardless of the billing question, why are there 2-3 instances? What's happening here is that the loading requests for this app have an average latency of 10.6s: https://appengine.google.com/logs?app_id=jadsbeta&version_id=29.353002800509795229&severity_level_override=1&severity_level=3&tz=US%2FPacific&filter=loading_request%3D1&filter_type=regex&date_type=now&date=2011-09-05&time=13%3A00%3A45&limit=20&view=Search. So, whenever the apps needs to do a loading request, this is problematic because, as that instance is conducting the loading request, there are more requests coming in, and those need to be serviced. But they can't be serviced by the existing instance, because it is doing a loading request (warm (non-loading) requests can be concurrent, but the loading request cannot be concurrent with non-loading-requests). So the scheduler turns up a new instance. [Note: the setting of min-pending-delay=15s actually should help in most cases here since loading request latency is still <15s. Unfortunately there is a bug where right now we are basically treating min-pending-delay=15s as equivalent to min-pending-delay=10s: http://code.google.com/p/googleappengine/issues/detail?id=5765. I have since fixed this and it should be live in a few weeks.] The recommendation to keep the clone running is not altogether a bad one, in that it means you have fewer loading requests, so this situation is avoided, as the warm request latency is nice and low. But if you can reduce his loading request latency, that would also help. But at the end of the day, it won't change the bill if you have set max-idle-instances. Afaict, 30 cents per day is probably the lowest your bill could get until the aforementioned bugfix is live. I hope that helps here, Jon On Sun, Sep 4, 2011 at 3:26 AM, Mike <[email protected]> wrote: > Like many, I'm deeply disappointed that Google went ahead with the > extortionate price increases, but instead of complaining about it > (because it seems to be falling on deaf ears) I'm trying to come up > with a workable solution to decrease costs. > > I have a couple of apps that see an average amount of traffic (around > 1 request every 3 seconds, fairly constantly) and I'm trying to > persuade the scheduler to keep only 1 instance running. > > I'm confident only 1 instance is needed. - Latency is around 150ms and > thread-safe is true. However, there always appears to be 2 or more > instances active. Its ridiculous to see these instances having served > 1 request sitting idle for 15 minutes. My understanding was that > enabling concurrent requests would stop these additional instances > starting up because if one request is waiting on I/O (common) another > request can be handled. > > What is the instance startup logic when thread-safe is true? I can not > see any difference. If anything, setting thread-safe to true has > increased the number of idle instances. In my eyes, the scheduler is > broken: > > Min Pending Latency appears to have no effect. Even set at 15s you > still see new instances start up unexpectedly. > > Max Idle Instances should drop down to zero. I do not want an > additional idle instance bringing the total instances to 2. > > Can we have an option to decrease the 15 minute idle time before > stopping. I do not want to pay for 15 minutes instance time to serve > one request. > > Another point: instance Startup time seems way too high in general. I > do not understand why it takes at least 4 seconds to start an > instance. Lightweight servlet containers can start in 200ms these > days. If startup time was optimised, the 15 minute idle period reduced > to say 5 minutes and the scheduler made less enthusiastic about > starting new instances I think the changes might be (almost) > acceptable. However, right now, I think a lot of people are most > annoyed about paying for instances that are sitting there (mostly) > idle - and this is understandable, we got used to paying for CPU time. > > I would really like to know how the scheduler decides whether or not > to start up a new instance when thread-safe is true. Is this > documented anywhere? > > Kind regards, > > Mike. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
