Oh, what I needed to mention is that if you're trying to time your app load with (say) a filter, a lot of this load time elapses before your code begins executing.
Jeff On Fri, Apr 20, 2012 at 11:44 AM, Jeff Schnitzer <[email protected]>wrote: > As far as anyone outside Google can tell, the biggest issue involves > reading files off the filesystem. This may have changed recently since > it's been a while since I (or any of the other users who commonly post > about such stuff) did any measurements, but reading files seems to be > painfully slow. > > Some things to consider: > > * Tools that do classpath scanning (Spring, Resteasy, JDO, etc) open > every jar and class file looking for annotations. These are usually the > biggest enemy. If you can disable scanning, it will usually help a lot - > although this means more manual configuration. > > * Lots of little files take longer than a few big ones. I found a small > but significant improvement (maybe 20%) by jaring up my class files - not > enough that I actually still do it. The numbers might be different if you > use classpath scanning. > > * Consider carefully each jar in your project and whether you really need > it. > > Jeff > > > > On Fri, Apr 20, 2012 at 3:34 AM, Eliran Bivas <[email protected]>wrote: > >> Just to add some information: >> When timing the load time of my app (not from the dev server startup but >> starting from the point my code is beging to load) it takes my application >> less than 8 seconds to load (few cases were 5secs). So I still can't figure >> out why on AppEngine it can sometimes take more than 60 seconds... >> >> I'd appreciate some analysis guidelines so I can further investigate what >> makes the app (sometimes) load very slowly. >> >> >> On Monday, April 16, 2012 10:13:50 AM UTC+3, Eliran Bivas wrote: >>> >>> Thanks for the clarification, it should be part of the documentation for >>> AppEngine architecture. >>> >>> As for my question, I believe that a Maven multi-module Spring >>> application has different view on loading times. >>> In earlier posts here it was mentioned that loading several JARs might >>> be an issue. BUT, that's how Maven works. >>> Even core Spring Framework consist of more than 10 JARs. >>> >>> Are there any recommendation on what to look for (analysis tools would >>> be great) in such deployment? Or maybe how such configuration should behave >>> (prefer lazy bean initialization over pre-init singletons)? What about the >>> JARs, would flatten them to a single Uber-JAR will help? And again, my >>> context loading is without DB operations or any HTTP connection. I assume >>> that as my project gets larger, and additional JARs will be added - are >>> there any best practices for such scenario also? >>> >>> On Monday, April 16, 2012 2:53:14 AM UTC+3, Jeff Schnitzer wrote: >>>> >>>> Woah! 30s to start your app in your local dev environment? That's >>>> nuts. >>>> >>>> You have a mistaken perception of GAE. In nearly all respects, your >>>> local dev environment will perform a single thread of execution faster than >>>> production. Your local machine has dedicated CPU cores and I/O bandwidth, >>>> all local. It has a mock datastore which likely has no synchronization >>>> issues. >>>> >>>> In production, the filesystem is loading across a network. Nearly all >>>> service calls require an RPC to a remote machine somewhere else in the >>>> cluster - maybe to several machines. You're sharing CPU cores and RAM with >>>> a dozen other apps, some of which might be really busy. >>>> >>>> What you get in production is a system that is not especially fast but >>>> nearly always consistent no matter what the queries/sec or database size. >>>> >>>> Your best strategy is to figure out why your app is taking so long to >>>> start up and address that. >>>> >>>> Jeff >>>> >>>> On Fri, Apr 13, 2012 at 2:31 PM, Eliran Bivas >>>> <[email protected]>wrote: >>>> >>>>> Do you have any recommendations how to reduce the amount of IO >>>>> operations during startup? >>>>> My app uses Spring and like many other Java best practices followed, >>>>> my project consists of several Maven modules which results several jars >>>>> creation. >>>>> I do not load any data from datastore or require HTTP connection >>>>> during startup. >>>>> >>>>> I understand the AppEngine filesystem is extremely slow, but on my >>>>> local dev machine loading my app takes ~30sec, so I assume that in a much >>>>> superior infrastructure it should take even faster. I believe AppEngine >>>>> should provide some property to extend loading requests to reach over 60 >>>>> seconds (like cron operations are allowed 10mins runtime). >>>>> >>>>> Thanks in advance >>>>> >>>>> >>>>> >>>>> On Friday, April 13, 2012 5:51:15 PM UTC+3, Jeff Schnitzer wrote: >>>>>> >>>>>> Right. The problem is the 61244ms that it takes to start your app. >>>>>> How long does it normally take? >>>>>> Look at past warmup requests (the ones that work) and see how long >>>>>> they take. My guess is that >>>>>> the number is close to 60s. If GAE gets marginally slower, it pushes >>>>>> you over the edge. >>>>>> >>>>>> As for why your app takes so long to startup, I can't begin to >>>>>> speculate. What does it do at startup? >>>>>> Do any of your frameworks do classpath scanning? Do you have a lot >>>>>> of big jars? Zillions of >>>>>> class files? Do you load data from the datastore or blobstore? >>>>>> >>>>>> You need to diagnose the warmup requests that *do* work. Sure, look >>>>>> at appstats - although >>>>>> that will only show issues if your warmup fetches data from services >>>>>> (ie, not the filesystem). >>>>>> However, you really should be able to think about it a few minutes >>>>>> and figure out the problem. >>>>>> App startup is almost 100% deterministic. There are no parameters. >>>>>> You know what it does. >>>>>> >>>>>> Jeff >>>>>> >>>>>> On Fri, Apr 13, 2012 at 9:55 AM, Rui Oliveira >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> Hi Jeff >>>>>>>> >>>>>>>> Thanks for your replay. Your answer was very important for me to >>>>>>>> start looking to the right part of the problem. >>>>>>>> >>>>>>>> Just to clarify :"startup time" is the time to start a new >>>>>>>> instance right? >>>>>>>> >>>>>>>> How can I analyze the startup time inside the server? I'm looking >>>>>>>> to the appengine logs, appstats, and speedtracer, but in none of them >>>>>>>> I can >>>>>>>> find whats happening during the startup. >>>>>>>> >>>>>>>> If I deploy the program I'm getting this kind of logs: >>>>>>>> >>>>>>>> >>>>>>> 1. 012-04-13 11:33:35.889 /_ah/warmup 500 6******1244ms 0kb >>>>>>> 2. W2012-04-13 11:33:35.787 EXCEPTION com.google.apphosting.api.* >>>>>>> *Dead****lineExceededException: This request (08f66682e6ba5919) >>>>>>> started at 2012/04/13 11:32:35.741 UTC and was still e >>>>>>> 3. E2012-04-13 11:33:35.790 javax.servlet.**Ser****vletContext >>>>>>> log: unavailable javax.servlet.**UnavailableExcep****tion: This >>>>>>> request (08f66682e6ba5919) started at 2012/04/13 11:32:35.741 U >>>>>>> 4. W2012-04-13 11:33:35.810 Failed startup of context >>>>>>> com.google.apphosting.utils.**je****tty.** >>>>>>> RuntimeAppEngineWebAppCont****ext@**1811e2c{/,/base/data/home/*** >>>>>>> ***apps/s~airmenudemo/29.**35817792****1953 >>>>>>> 5. C2012-04-13 11:33:35.816 Uncaught exception from servlet >>>>>>> javax.servlet.**UnavailableExcep****tion: Initialization failed. >>>>>>> at com.google.apphosting.runtime.******jetty.AppVersionHandlerMa >>>>>>> 6. I2012-04-13 11:33:35.819 This request caused a new process to >>>>>>> be started for your application, and thus caused your application >>>>>>> code to >>>>>>> be loaded for the first time. This requ >>>>>>> 7. W2012-04-13 11:33:35.819 A problem was encountered with the >>>>>>> process that handled this request, causing it to exit. This is >>>>>>> likely to >>>>>>> cause a new process to be used for the nex >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On appstats: ( http://www.airmenudemo.**appsp**** >>>>>>>> ot.com/appstats/stats<http://www.airmenudemo.appspot.com/appstats/stats> >>>>>>>> ) >>>>>>>> >>>>>>>> (16) 2012-04-13 11:36:31.587 "GET /appstats/" 307 >>>>>>>> <http://www.airmenudemo.appspot.com/appstats/details?time=1334316991587>real=215ms >>>>>>>> api=0ms overhead=0ms (0 RPCs) >>>>>>>> >>>>>>>> (17) 2012-04-13 11:35:49.207 "GET /symbolmanifest.json" 404 >>>>>>>> <http://www.airmenudemo.appspot.com/appstats/details?time=1334316949207>real=609ms >>>>>>>> api=0ms overhead=0ms (0 RPCs) >>>>>>>> >>>>>>>> (18) 2012-04-13 11:27:25.255 "GET /_ah/warmup" 200 >>>>>>>> <http://www.airmenudemo.appspot.com/appstats/details?time=1334316445255>real=441ms >>>>>>>> api=0ms overhead=0ms (0 RPCs) >>>>>>>> >>>>>>>> As you can see after deploy the appstats don't log nothing. >>>>>>> >>>>>>> After deploy I can't even open appstats. >>>>>>> >>>>>>> After some minutes / hours everything starts ok. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Rui >>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Google App Engine" group. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/**ms**g/google-appengine/-/frES_**he8D** >>>>>>> HYJ<https://groups.google.com/d/msg/google-appengine/-/frES_he8DHYJ> >>>>>>> . >>>>>>> >>>>>>> To post to this group, send email to google-appengine@googlegroups.* >>>>>>> ***com <[email protected]>. >>>>>>> To unsubscribe from this group, send email to >>>>>>> google-appengine+unsubscribe@**g**ooglegroups.com<google-appengine%[email protected]> >>>>>>> . >>>>>>> For more options, visit this group at http://groups.google.com/** >>>>>>> group**/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>>>>>> . >>>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To view this discussion on the web visit https://groups.google.com/d/* >>>>> *msg/google-appengine/-/**WJ0eGsUpEgwJ<https://groups.google.com/d/msg/google-appengine/-/WJ0eGsUpEgwJ> >>>>> . >>>>> >>>>> To post to this group, send email to google-appengine@googlegroups.** >>>>> com <[email protected]>. >>>>> To unsubscribe from this group, send email to >>>>> google-appengine+unsubscribe@**googlegroups.com<google-appengine%[email protected]> >>>>> . >>>>> For more options, visit this group at http://groups.google.com/** >>>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>>>> . >>>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/edj6rgd5s54J. >> >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
