For the request you describe here, what modules are you importing? Are the Gets by keyname or are they filtered queries?
Are you doing the puts all at once (for Python, that would mean putting all the entities into a entityList and calling db.put(entityList)) or are you doing separate puts for each entity? Have you looked into using memcache for the gets? (if you can't use memcache, why not? etc) Are the CPU Times you are reporting the average for a cold instance of the request or a hot instance (If you look in the logs for the request, it should tell you "This request caused a new process to be started.." if it is a cold instance)? How many indexes are defined on the Model types that are being put to the datastore? Is there any way around using a Transaction? (not really worth exploring unless you've tested the process without a transaction.. and it performs drastically better) Have you looked into manually logging cpu time in the code? (e.g. recording cpu time before the gets.. and then after.. to see what CPU they use.. then doing the same for the Puts) On Wed, Sep 29, 2010 at 5:04 PM, Michael <[email protected]> wrote: > I'm using GAE as the backend for a game and I'm looking for some help in > optimizing my CPU usage. With my current usage and at the current billable > rates, it looks like I will not be able to scale to the hundreds of > thousands, let alone millions, of transactions a day without spending a ton > of money. I've found CPU Usage to be the most frustrating quota to wrap my > head around since it is the hardest to track and profile and isn't under the > full control of the application developer. Anyway, with that preface, I > want to give out a couple numbers to see if these sound reasonable. > > My most expensive request has the following stats (I estimate 5-10 of these > per user per day): > > > - Avg. Realtime : 200ms > - Avg. CPU Time: 650ms > - Avg. API Time: 500ms > - Avg. # RPCs : 9-11 > - 3 Gets (retrieving about 1KB) > - 1 BeginTransaction > - 4-6 Puts (adding about 256 bytes) > - 1 Commit > > I have used Appstats, and it showed that previously I was being hit by a > larger number of RPCs. That has been reduced drastically and improved the > performance a lot. Now that I have the number of RPCs down to as little as > possible, performance still isn't where I need it. Realtime performance is > pretty good, but the CPU and API times are still way too high. > Unfortunately Appstats for Java doesn't give me a view of CPU/API time. So > here are some questions I have, but any general guidance is appreciated: > > > 1. Do the above times seem reasonable for the number and type of RPCs? > What other information should I look at or provide to get a good view of > the > RPC/non-RPC CPU costs? > 2. Will Java Appstats be providing CPU/API breakdowns at some point > soon? > 3. Should I consider changing to Python for better Appstats or better > overall performance? What are the average performance gains for Python > over > Java? > 4. Does anyone else use GAE this heavily (i.e. upwards of a million > transactions a day with non-trivial CPU usage)? > 5. What additional profiling/optimization steps should I consider? > cProfile? Sprinkling QuotaService calls throughout the app? > > > Thanks for the help! > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
