great tips. thanks so much. I'm using GAE1.3.1 and will try the cursor for reading data. yes, I get lots of "com.google.apphosting.runtime.HardDeadlineExceededError" because the 30sec limit. Will try to re-engineer to make each task smaller.
-aj On Mon, Feb 22, 2010 at 10:31 AM, Marc Provost <[email protected]> wrote: > Hi AJ, > > Here's a few tips from my experience: > > * If your application currently does not have much traffic, most of > your requests will be "loading requests". This could explain the > randomness your are seeing. The app engine needs to load and prepare > your application to handle a request. In my application (java), a > loading request takes around 2-6 seconds. Once your application is > loaded, it will stay in memory for some time and subsequent requests > will be "normal requests". See > http://googleappengine.blogspot.com/2009/12/request-performance-in-java.html > for more info. Also, make sure you are using the most recent version > of the app engine so that pre-compilation is enabled by default. I > have not tried the python implementation of the app engine yet, but I > heard that it process loading requests faster. > > * Once my application is loaded, fetching the 20 first results of a > simple query takes at most 1-2 seconds, most of the time less than a > second. I use memcache to cache the derived product (usually an html > page) of common requests. Once a request is cached, it will take > 100-200ms to serve. > > * When you list the topics, make sure you are using cursors: > > http://code.google.com/appengine/docs/java/datastore/queriesandindexes.html#Query_Cursors > > * When you fetch feeds, you will need to split the work in very small > chunks because of the 30 seconds limit. When you say "fails randomly", > are you getting > "com.google.apphosting.runtime.HardDeadlineExceededError"? From my > experience, appengine behave much better when each of your tasks write > to very few entities. In my application, I'm being very strict: each > task writes to at most one entity. For example, I also need to parse > an external feed and then update 1000 entities with the data from that > feed. In order to achieve that efficiently, I spawn a thousand tasks, > each one updating only one entity. > > Hope it helps, > Marc > > > On Feb 21, 7:02 pm, AJ Chen <[email protected]> wrote: > > I like the potential of appengine as cloud computing platform. Using > > eclipse, I can code in eclipse and then one-click to deploy the changes > to > > production. It's awesome! It's also an exciting experience to learn the > new > > programing patterns GAE requires, such as task queue, object data store, > > mencache. All these new stuff are fine as long as it delivers the > > performance at the end. After running my apphttp://realmon9.appspot.comon > > production for a while, I found the response time very often is too long, > > 5-20sec, in the unusable range. Of course, the performce depends on how > > complicated it is to generate the response per request. I'm going to give > a > > very brief description of a typical request, and appreciate your > suggestion > > for improving the performance.. > > > > My app "realmon9" <http://realmon9.appspot.com> is a social media > monitoring > > application designed as a component in the google cloud so that it can be > > connected to enterprise CRM like salesforce. It basically allows > > organization to monitor a large number of topics on social media and > brings > > the relevant conversations/leads to CRM for marketing/PR/support/research > > purpose. The topics, twiter conversations and blogs are stored in > datastore > > and the operation is quite simple and straightforward. For example, > listing > > a list of topics or listing conversations (20 per page) for a topic. I > > expect this type of viewing request takes <1 sec to respond. It requires > > querying 2-4 kinds of data per request and there are only small number of > > data in this initial stage. But, very often it takes 10 second to respond > to > > a simple request. I'm using java and JDO to query datastore. I have not > > done anything to customize the index configuration yet. Where should I > look > > for performance optimization? > > > > I also use task queue to fetch feeds in the background. Because the > reponse > > is slow, a large percentage of simple feed fatch tasks fail randomly. > > > > One observation: viewing the same page (e.g. listing topics) sometimes > takes > > no time, but sometimes takes 10 second. it's all random, which is > probably > > due to the distributed nature of GAE. It may be hard to figure out what > to > > improve on the app side when GAE varies wildly in terms response time. > > Anybody know the expected response time or behavior from GAE? > > > > This google app is porting from my server application onhttp:// > web2express.org. I can make the responce on regular tomcat server > > fast, but GAE is the uncharted torritory. I"m still learning and looking > for > > best practice ideas. > > > > thanks, > > -aj > > -- > > AJ Chen, PhD > > Chair, Semantic Web SIG, sdforum.orghttp://web2express.org > > @web2express on twitter > > Palo Alto, CA, USA > > 650-283-4091 > > *Monitoring social media in real time* > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org @web2express on twitter Palo Alto, CA, USA 650-283-4091 *Monitoring social media in real time* -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
