I used to have an API app running at 1700 req/s with spikes about 2000+ running in multithreaded mode with about 50 instances in parallel. Not bad but costs where spiking for some reason. After investigation it seemed that memcache wasnt holding the data long enough between requests and the app had to fetch from the datastore too frequently. Also if you look at the required memory we were talking several TB although the app itself do not need that much
Even after resorting to multi fetch/put with memcache and datastore and rewriting the entire code to get rid of JDO and only rely on low level API (Which i strongly recommend btw it was the best move i ever did with GAE) average response time was about 500ms which wasnt too bad given the load but still not good enough given the task each request had to to (basically just reading some data from the cache and once every N req per client compute some stats). Costs where about 1000$/month and relying on master/slave DS (i can imagine the costs on HR would have been much higher) I finally gave up on GAE for that kind of basic task with high polling frequency and built my own config of NGINX + Php-FPM + APC +MySQL on unix sockets and one single AWS high cpu instance can handle 4000+ req/s for about 300$ a month. the server will store the nth requests in mysql and a shell script will report the batch to the main GAE app periodically to be processed by long running tasks to compute the stats. Average response time of the above config is 55ms... i built an image of this config and intend to use it on other projects as well. Thus said i love GAE i can sleep in peace knowing all my apps are running fine and can handle huge traffic (over 3Billions queries a month in that case). But at some points the costs are growing fast as you can less and less rely on memcache. the current nginx box only uses 256MB of memory and the cache never ejects any entry. Compare that to the N TB that gae claimed using there should be a lot of margin for improvement and maybe letting us have more control over the size of memcache. i can share more details with the GAE team/ikai if needed but cannot detail on this group these API apps not being public access. On May 1, 1:51 am, "Brandon Wirtz" <[email protected]> wrote: > Two of our deployments cruise along at 60 request per second 12-14 hours a > day every day. And site at about 30 instances all of that time. Thus far > running on HR they have had Zero down time. > > From: [email protected] > [mailto:[email protected]] On Behalf Of Jay > Sent: Saturday, April 30, 2011 8:15 AM > To: [email protected] > Subject: [google-appengine] Re: GAE Hauling the Freight > > I had a similar experience this month. Due to advertising related to a movie > release. Requests/sec. spiked up to 50 per second and held there for a few > hours. Absolutely no problem on app engine. We didn't have to touch > anything. (of course we had thought about it before and maintained a very > light weight approach). But operationally, we could just watch it go. GAE > spun up 29 instances to handle the traffic at that point if memory serves. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
