On Mon, Jul 25, 2011 at 9:25 PM, Galoch <[email protected]> wrote: > > Hi Johan, > > Thanks for following up on this discussion. > > I have new findings to share with you since GAE now works a bit > differently with all these recent changes. Before I do that I will > share the changes I made to our app which has helped us mitigate some > performance issues. > > 1. We moved all our cron jobs to the Backends with 1 B1 RI. Amazingly > it is able to handle all the load single handed compared to 6 > instances (3 RI + 3 DI) in the regular app. We have our eye brows > raised! But we are playing it with caution by setting <max-concurrent- > requests> to 1. Will gradually raise the limits until it starts > complaining. > > 2. We set Max Idle Instances to 6 and Min Pending Latency to 15s. I > recommend keeping Max Idle Instances to at least 4. Because if you > keep it at 3 GAE kills and creates DI "very" aggressively ... in some > cases 10-20 seconds.
Thanks for investigating those behaviors, feel free to continue to share your findings with the community. > 3. In our regular app we are running dummy cron jobs to keep 3 DI up > at all times. Again, amazingly with many concurrent users they are > able to handle ALL the load. On the contrary RI are almost untouched. > Here's the latest statistics: > > QPS* Latency* Requests Errors Age Memory > Availability > 0.000 0.0 ms 25 0 12:43:22 82.3 MBytes > Resident > Icon Resident > 0.000 0.0 ms 10 0 12:44:45 80.6 MBytes > Resident > Icon Resident > 0.000 0.0 ms 5 0 12:44:46 81.9 MBytes > Resident > Icon Resident > 0.350 19.1 ms 11762 0 9:45:11 118.5 MBytes > Dynamic Icon > Dynamic > 0.317 17.5 ms 3890 0 3:11:04 97.7 MBytes > Dynamic Icon > Dynamic > 0.333 67.5 ms 15604 0 12:43:41 104.8 MBytes > Dynamic Icon > Dynamic > > But this is just a sample set of users. I don't know yet if we can > scale on this principle but it seems to work on light load. But this > is definitely short term fix since this is going to be expensive under > new billing. I believe those stats illustrate the rule we discussed before: Idle DI > Idle RI. Keep in mind that under the new pricing model Min Idle Instances should superseed Always On instances, as described by gregd in the billing faq. So hopefully you should not have idling DI instance wandering around, and you will be able to control this with Max Idle Instances anyway. > After spending fairly good amount of time collecting these statistics > I can boil down to a few things that I want to share, that can help us > tune our app to GAE. > A. Know the current usage of an instance (RI or DI) in terms of > current CPU utilization in percentage. This will help us correlate CPU > utilization shown in each request in the logs and help us tune our > queries. Right now there is no way to determine what effect the tuning > had on an individual instance. > > B. Know the number of current threads running within an instance > (maybe in a span of 15 seconds or so). This will help us identify > ratio of thread handling capacity vs CPU consumed (point A). Higher > the number means better throughput. Lower the number means we need to > closely examine our requests. > > C. Able to override the 30 second deadline for the warm up requests so > that we have enough time to load our frameworks. > > D. Able to specify how long an instance can live before reclaimed by > GAE. Something similar to session timeouts. This will help avoid > unnecessary warm up requests and will improve overall performance. > > E. Ability to specify rules within GAE console that will adjust the > Max Idle Instances during different times of a day and week. > > F. Able to specify the size of DI (something similar to Backends B1, > B2 ... so on) instead of standard DI. Should be able to configure > along with each rule mentioned in point E. > > > I understand that C, D, E, F may involve some additional costs to us > but these short / long term enhancements will hugely help us run our > business applications on GAE. The idea is to give us more time and > capacity with the DI while keeping a control on the cost. I think your suggestions would be a great addition to the admin-console, and let user have more visibility and control over their instances and thus over billing. You should definitely fill separate feature requests on the public issue tracker for them. In the meantime, I will make sure the engineering team is aware of your suggestion. > Let me know if I missed something that is already there or I am going > off tangent here. You're completely on the topic of this discussion :) > On Jul 25, 9:18 am, Johan Euphrosine <[email protected]> wrote: >> Hi Francois, >> >> I think the help text of the 'Idle Instances' settings is pretty >> self-explanatory: >> """ >> You will not be charged for instances over the specified maximum. >> """ >> >> I can investigate on why these instances are created even thought you >> setup a high min pending latency, and a a Max idle instances >> corresponding to the number of Always On instance. >> >> Feel free to open a Production issue, with your application id if this >> is affection your operation. >> >> Thanks in advance. >> >> >> >> >> >> >> >> >> >> On Mon, Jul 25, 2011 at 4:58 PM, Francois Masurel <[email protected]> >> wrote: >> > Something strange : >> > GAE keeps starting new dynamic instances though I have set the Min Pending >> > Latency to 15s via the Application Settings page. >> > My requests are served in a few ms on average and I have >> > multithreading enabled. >> > These instances are immediately destroyed as I have set the Max Idle >> > Instances to 3 (corresponding to my always on instances). >> >> > But still lots of warmup requests in my logs showing these dynamic >> > instances >> > start every few minutes. >> > Will I have to pay for all these short lived dynamic instances ? >> > My app ID is : vncts1 >> > Thanx for your help. >> > Francois >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To view this discussion on the web visit >> >https://groups.google.com/d/msg/google-appengine/-/TVr3Ko1-baIJ. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> >http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> Johan Euphrosine (proppy) >> Developer Programs Engineer >> Google Developer Relations > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
