Hi All, First, thanks for all of the feedback, it's very helpful for us to hear directly from customers about how issues affect them and what would make them feel more confident using the platform. Please feel free to reach out to me directly (cilvento@) if you have anything that you'd like to discuss without an audience.
For this issue in particular, we are investigating the duration and severity of the incident on our side. No one can promise a 100% bug-free environment, but we do try to be as transparent as possible about these issues, and we'll be posting a more detailed report soon. Thanks, Christina On Thu, Sep 13, 2012 at 8:25 AM, hyperflame <[email protected]> wrote: > Christina, > > Let me take a wild guess at what happened internally. A few months > ago, there was a 100+ post thread on this mailing list complaining > about long instance startup times (specifically, that classloading was > slow). You guys had an internal discussion, and produced some code to > streamline classloading, and it's failing because the streamlined > classloading process made some assumptions that don't hold up in > production. > > Let me make an educated guess at why the failure is happening. Jeff > Schnitzer reported that the base exception was > "java.lang.ClassNotFoundException: java.io.FileOutputStream". But GAE > shouldn't need to load FileOutputStream, since the GAE JRE whitelist > (located at https://developers.google.com/appengine/docs/java/jrewhitelist > ) doesn't even include FileOutputStream. There is no legitimate reason > to load or make available FileOutputStream if we're not allowed to use > it. So why is GAE attempting to load a non-whitelisted class? > > My guess is that GAE deployed with the wrong copy of the SDK; perhaps > there was a reference to FileOutputStream in one of the internal GAE > classes, and everything went kaput when the servers failed to find the > reference. Or perhaps there was a last minute change by a programmer > who forgot that FileOutputStream wasn't available in GAE. > > Am I close or completely wrong? > > On Sep 12, 4:56 pm, Christina Ilvento <[email protected]> wrote: > > *Hi All, > > > > Beginning yesterday, September 11, Google App Engine experienced two > > periods of serving degradation for a subset of Java applications due to a > > gradual roll-out of a new version of the Java runtime. Affected > > applications would have seen errors related to class loading. We have > > resolved the first issue by fixing the underlying bug. We are still > > investigating the cause of the second issue but have rolled back the > > problematic update and all affected applications should now be returned > to > > normal serving behavior. > > > > No changes to your code or application configuration are needed at this > > time. We apologize for any inconvenience this issue has caused, and we’ll > > follow up with more details on the underlying incident and resolution > soon. > > > > Regards, > > Christina Ilvento, App Engine PM* > > > > > > > > On Wed, Sep 12, 2012 at 1:34 PM, Kaan Soral <[email protected]> wrote: > > > I wonder what happened, subscribed to this topic to be updated, hope > > > someone explains what happened and additionally hope this never > happens to > > > me (python) :) > > > > > On Wednesday, September 12, 2012 11:00:49 PM UTC+3, Jeff Schnitzer > wrote: > > > > >> We are back up and running now after 2 hrs of downtime. > > > > >> To whoever fixed it: THANK YOU!!! > > > > >> To whoever broke it in the first place: SPANKINGS!!! > > > > >> Jeff > > > > >> On Wed, Sep 12, 2012 at 12:11 PM, Jeff Schnitzer < > [email protected]> > > >> wrote: > > >> > More information: > > > > >> > * The failure began at 10:54am (pacific). > > >> > * Same app on different appid has the same problem. > > > > >> > (as listed in the stacktraces, the appid is voost0) > > > > >> > Jeff > > > > >> > On Wed, Sep 12, 2012 at 11:53 AM, Jeff Schnitzer < > [email protected]> > > >> wrote: > > >> >> HEEEEEEEEEEEEELP! > > > > >> >> We have tried everything at this point. Shut down instances, > tried to > > >> >> deploy a new version, even tried old versions. We've reported a > > >> >> production issue. Something is broken inside of GAE. The Guice > error > > >> >> must be a symptom; the smoking gun seems to be: > > > > >> >> java.lang.**ClassNotFoundException: java.io.FileOutputStream > > > > >> >> We've been down for 30 mins and getting complaints from our > clients. > > >> >> This looks really, really bad. It's my nightmare scenario - an > outage > > >> >> in GAE that is small enough not to raise major alarm bells, but > > >> >> nevertheless cripples my business. It is not the first time this > has > > >> >> happened. It is shaking my faith in GAE. > > > > >> >> Jeff > > > > >> >> On Wed, Sep 12, 2012 at 11:11 AM, Jeff Schnitzer < > [email protected]> > > >> wrote: > > >> >>> Our app has been running fine on the same version, no code deploys > > >> >>> (our versions are timestamped so this is 100% certain), for two > days. > > >> >>> All of a sudden (10 mins ago) our app stopped running. Every > attempt > > >> >>> to run a request produces this cryptic message: > > > > >> >>> -------- > > > > >> >>> 2012-09-12 10:56:21.791 > > >> >>> com.google.inject.servlet.**GuiceFilter setPipeline: Multiple > > >> Servlet > > >> >>> injectors detected. This is a warning indicating that you have > more > > >> >>> than one GuiceFilter running in your web application. If this is > > >> >>> deliberate, you may safely ignore this message. If this is NOT > > >> >>> deliberate however, your application may not work as expected. > > >> >>> D 2012-09-12 10:56:21.792 > > >> >>> st.voo.tick.util.cambridge.**CambridgeSetup <init>: Establishing > > >> >>> cambridge view resolver > > >> >>> I 2012-09-12 10:56:21.792 > > >> >>> st.voo.tick.GuiceConfig contextInitialized: Guice initialization > took > > >> 514 millis > > >> >>> W 2012-09-12 10:56:21.888 > > >> >>> Failed startup of context > > >> >>> > com.google.apphosting.utils.**jetty.**RuntimeAppEngineWebAppContext@* > > >> > *1479784{/,/base/data/home/**apps/s~voost0/2012-09-10-1715.**361669184733923098} > > > > >> >>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to > > >> >>> instantiate MessageBodyReader > > >> >>> at > org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** > > >> register(RegisterBuiltin.java:**35) > > >> >>> at > org.jboss.resteasy.spi.**ResteasyDeployment.start(**ResteasyDeployment.java:211) > > > > >> >>> at org.jboss.resteasy.plugins.**server.servlet.** > > >> > ServletContainerDispatcher.**init(**ServletContainerDispatcher.**java:67) > > > > >> >>> at org.jboss.resteasy.plugins.**server.servlet.** > > >> FilterDispatcher.init(**FilterDispatcher.java:39) > > >> >>> at > st.voo.tick.util.**GuiceResteasyFilterDispatcher.**init(** > > >> GuiceResteasyFilterDispatcher.**java:48) > > >> >>> at > com.google.inject.servlet.**FilterDefinition.init(**FilterDefinition.java:114) > > > > >> >>> at com.google.inject.servlet.**ManagedFilterPipeline.** > > >> initPipeline(**ManagedFilterPipeline.java:98) > > >> >>> at > com.google.inject.servlet.**GuiceFilter.init(GuiceFilter.* > > >> *java:172) > > >> >>> at > org.mortbay.jetty.servlet.**FilterHolder.doStart(**FilterHolder.java:97) > > > > >> >>> at > org.mortbay.component.**AbstractLifeCycle.start(**AbstractLifeCycle.java:50) > > > > >> >>> at > org.mortbay.jetty.servlet.**ServletHandler.initialize(**ServletHandler.java:662) > > > > >> >>> at > org.mortbay.jetty.servlet.**Context.startContext(Context.* > > >> *java:140) > > >> >>> at > org.mortbay.jetty.webapp.**WebAppContext.startContext(**WebAppContext.java:1250) > > > > >> >>> at > org.mortbay.jetty.handler.**ContextHandler.doStart(**ContextHandler.java:517) > > > > >> >>> at > org.mortbay.jetty.webapp.**WebAppContext.doStart(**WebAppContext.java:467) > > > > >> >>> at > org.mortbay.component.**AbstractLifeCycle.start(**AbstractLifeCycle.java:50) > > > > >> >>> at com.google.apphosting.runtime.** > > >> > jetty.AppVersionHandlerMap.**createHandler(**AppVersionHandlerMap.java:219) > > > > >> >>> at com.google.apphosting.runtime.** > > >> > jetty.AppVersionHandlerMap.**getHandler(**AppVersionHandlerMap.java:194) > > >> >>> at com.google.apphosting.runtime.**jetty.** > > >> > JettyServletEngineAdapter.**serviceRequest(**JettyServletEngineAdapter.**java:134) > > > > >> >>> at com.google.apphosting.runtime.** > > >> JavaRuntime$RequestRunnable.**run(JavaRuntime.java:447) > > >> >>> at > com.google.tracing.**TraceContext$**TraceContextRunnable.* > > >> *runInContext(TraceContext.**java:452) > > >> >>> at com.google.tracing.**TraceContext$** > > >> TraceContextRunnable$1.run(**TraceContext.java:459) > > >> >>> at > com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) > > > > >> >>> at com.google.tracing.**TraceContext$** > > >> > AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) > > > > >> >>> at com.google.tracing.**TraceContext$** > > >> > AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) > > > > >> >>> at com.google.tracing.**TraceContext$** > > >> TraceContextRunnable.run(**TraceContext.java:456) > > >> >>> at com.google.apphosting.runtime.** > > >> ThreadGroupPool$PoolEntry.run(**ThreadGroupPool.java:251) > > >> >>> at java.lang.Thread.run(Thread.**java:679) > > >> >>> Caused by: java.lang.RuntimeException: Unable to instantiate > > >> MessageBodyReader > > >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** > > >> registerProvider(**ResteasyProviderFactory.java:**761) > > >> >>> at > org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** > > >> registerProviders(**RegisterBuiltin.java:70) > > >> >>> at > org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** > > >> register(RegisterBuiltin.java:**31) > > >> >>> ... 27 more > > >> >>> Caused by: java.lang.SecurityException: Unable to get members for > > >> >>> class org.jboss.resteasy.plugins.**providers.DataSourceProvider > > >> >>> at > com.google.appengine.runtime.**Request.process-**b6ca2b194d66ed23(Request.java) > > > > >> >>> at java.lang.Class.**getConstructors(Class.java:**291) > > >> >>> at org.jboss.resteasy.util.**PickConstructor.** > > >> pickSingletonConstructor(**PickConstructor.java:27) > > >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** > > >> getProviderInstance(**ResteasyProviderFactory.java:**1032) > > >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** > > >> addMessageBodyReader(**ResteasyProviderFactory.java:**478) > > >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** > > >> registerProvider(**ResteasyProviderFactory.java:**757) > > >> >>> at > org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** > > >> registerProviders(**RegisterBuiltin.java:70) > > >> >>> at > org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** > > >> register(RegisterBuiltin.java:**31) > > >> >>> at > org.jboss.resteasy.spi.**ResteasyDeployment.start(**ResteasyDeployment.java:211) > > > > >> >>> at org.jboss.resteasy.plugins.**server.servlet.** > > >> > ServletContainerDispatcher.**init(**ServletContainerDispatcher.**java:67) > > > > >> >>> at org.jboss.resteasy.plugins.**server.servlet.** > > >> FilterDispatcher.init(**FilterDispatcher.java:39) > > >> >>> at > st.voo.tick.util.**GuiceResteasyFilterDispatcher.**init(** > > >> GuiceResteasyFilterDispatcher.**java:48) > > >> >>> at > com.google.inject.servlet.**FilterDefinition.init(**FilterDefinition.java:114) > > > > >> >>> at com.google.inject.servlet.**ManagedFilterPipeline.** > > >> initPipeline(**ManagedFilterPipeline.java:98) > > >> >>> at > com.google.inject.servlet.**GuiceFilter.init(GuiceFilter.* > > >> *java:172) > > >> >>> at > org.mortbay.jetty.servlet.**FilterHolder.doStart(**FilterHolder.java:97) > > > > >> >>> at > org.mortbay.component.**AbstractLifeCycle.start(**AbstractLifeCycle.java:50) > > > > >> >>> at > org.mortbay.jetty.servlet.**ServletHandler.initialize(**ServletHandler.java:662) > > > > >> >>> at > org.mortbay.jetty.servlet.**Context.startContext(Context.* > > >> *java:140) > > >> >>> at > org.mortbay.jetty.webapp.**WebAppContext.startContext(**WebAppContext.java:1250) > > > > >> >>> at > > > > ... > > > > read more »- Hide quoted text - > > > > - Show quoted text - > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Christina Ilvento | Google App Engine | [email protected] | (650)-201-9399 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
