Here's some more information: I have two environments, a production environment and a sandbox environment (same code, different appids). Production loading requests usually take ~50 seconds, with occasional (rare) 25s loads. Sandbox consistently takes 25s, with *very* rare numbers higher.
I log a few timing metrics at startup. Behavior is consistent with slow classloading, but I can't think of a good way to be certain. Is it possible to instrument the classloader and get load timings for specific classes? I've optimized my app in all the obvious ways: Lazily load everything that can be, eliminate any classpath scanning, using Objectify, etc. However, it still requires a lot of classes to get my app started, and I don't see any way around it: * Objectify requires every class to be registered up front. This is taking 5-8 seconds to regster 36 entity classes (plus a fair bit of embedded structure). I am intimately familiar with what Ofy does during registration - this isn't a computational cost. * I'm using Guice (development mode), but this isn't really the core of the problem. The issue is that I'm using JAX-RS (Resteasy) to support my REST API. In order for the @Path annotations to be recognized, every resource class must be registered - and thus loaded. There's ~80 of these classes. The Guice initialization part (which includes Objectify registration) takes ~30s in production, ~15s in sandbox. I don't see an obvious way to optimize this short of going back to Java programming circa 2002. As long as my sitemap is defined by @Path annotations and not a bunch of text in an xml file, those classes are going to have to be loaded before my app starts. Entity classes must be registered and introspected before any data is loaded from the datastore. I'm not even asking for classpath scanning... I just want normal classes to load in reasonable time. This problem is going to get significantly worse over the next year. We add classes every day; our app gets more features, not fewer. I guess my next experiment will need to be JARing my WEB-INF/classes. Rafael: Thanks for the links... John Patterson's comment on David Chandler's article was interesting. I use Guice AOP in several rather important places; I really hesitate to remove it. At the very least it would require adding a lot of tedious, error-prone code to replace my interceptors. I would like to hear from someone who has measured the effects of stripping unused code out of third-party jars. Does that really help? It would be heinously complicated to maintain this, but I'm very concerned that I'm hitting a scalability limit of appengine - something like "your application cannot be more complex than N classes". Thanks, Jeff On Sun, Jun 17, 2012 at 4:46 AM, Per <[email protected]> wrote: > Hi Jeff, sounds awful. > > Even 20s on a good day is a lot IMO. We're using Wicket, which is on the > heavy side, and still the startup typically only takes 10 to 15s, and that's > including rendering an initial page too (so it basically involves firing up > all subsystems too, loading all classes, contacting the database several > times, filling some caches, etc). > > I'm guessing you're experiencing some "new release testing" by the Google > Team, so that might account for a certain slowdown. We've seen our average > latency increase from 250ms to 380ms over night, 2 days ago. Maybe your > slowdown is related to a similar service change. > > But it sounds like your application startup is too slow to begin with. Do > you have logging in there which can pinpoint what parts take how much time? > Would be interested to learn about that. > > Cheers, > Per > > > > > On Saturday, June 16, 2012 11:56:33 PM UTC+2, Jeff Schnitzer wrote: >> >> We're having a big problem with instance startup time. It varies >> between 20s and 60+ seconds, and lately it's tending towards the high >> end. We're starting to experience downtime because instances get >> deadlined before they go active. >> >> This app is well optimized for GAE. There's no classpath scanning and >> it doesn't try to eagerly load data. On a good day it starts in >> 20s... so at this point there's not really much I can do. >> >> I have a cron task that performs a db cleanup once a minute, and since >> crons can run over 60s I can eventually get one instance started, >> which is enough to serve traffic at the moment. But at this point I >> can no longer deploy code over old versions because the appserver >> restart will fail. >> >> Please help. >> >> Jeff > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/CzSFxHEhj3QJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
