*Hi All, Beginning yesterday, September 11, Google App Engine experienced two periods of serving degradation for a subset of Java applications due to a gradual roll-out of a new version of the Java runtime. Affected applications would have seen errors related to class loading. We have resolved the first issue by fixing the underlying bug. We are still investigating the cause of the second issue but have rolled back the problematic update and all affected applications should now be returned to normal serving behavior.
No changes to your code or application configuration are needed at this time. We apologize for any inconvenience this issue has caused, and we’ll follow up with more details on the underlying incident and resolution soon. Regards, Christina Ilvento, App Engine PM* On Wed, Sep 12, 2012 at 1:34 PM, Kaan Soral <[email protected]> wrote: > I wonder what happened, subscribed to this topic to be updated, hope > someone explains what happened and additionally hope this never happens to > me (python) :) > > On Wednesday, September 12, 2012 11:00:49 PM UTC+3, Jeff Schnitzer wrote: > >> We are back up and running now after 2 hrs of downtime. >> >> To whoever fixed it: THANK YOU!!! >> >> To whoever broke it in the first place: SPANKINGS!!! >> >> Jeff >> >> On Wed, Sep 12, 2012 at 12:11 PM, Jeff Schnitzer <[email protected]> >> wrote: >> > More information: >> > >> > * The failure began at 10:54am (pacific). >> > * Same app on different appid has the same problem. >> > >> > (as listed in the stacktraces, the appid is voost0) >> > >> > Jeff >> > >> > On Wed, Sep 12, 2012 at 11:53 AM, Jeff Schnitzer <[email protected]> >> wrote: >> >> HEEEEEEEEEEEEELP! >> >> >> >> We have tried everything at this point. Shut down instances, tried to >> >> deploy a new version, even tried old versions. We've reported a >> >> production issue. Something is broken inside of GAE. The Guice error >> >> must be a symptom; the smoking gun seems to be: >> >> >> >> java.lang.**ClassNotFoundException: java.io.FileOutputStream >> >> >> >> We've been down for 30 mins and getting complaints from our clients. >> >> This looks really, really bad. It's my nightmare scenario - an outage >> >> in GAE that is small enough not to raise major alarm bells, but >> >> nevertheless cripples my business. It is not the first time this has >> >> happened. It is shaking my faith in GAE. >> >> >> >> Jeff >> >> >> >> On Wed, Sep 12, 2012 at 11:11 AM, Jeff Schnitzer <[email protected]> >> wrote: >> >>> Our app has been running fine on the same version, no code deploys >> >>> (our versions are timestamped so this is 100% certain), for two days. >> >>> All of a sudden (10 mins ago) our app stopped running. Every attempt >> >>> to run a request produces this cryptic message: >> >>> >> >>> -------- >> >>> >> >>> 2012-09-12 10:56:21.791 >> >>> com.google.inject.servlet.**GuiceFilter setPipeline: Multiple >> Servlet >> >>> injectors detected. This is a warning indicating that you have more >> >>> than one GuiceFilter running in your web application. If this is >> >>> deliberate, you may safely ignore this message. If this is NOT >> >>> deliberate however, your application may not work as expected. >> >>> D 2012-09-12 10:56:21.792 >> >>> st.voo.tick.util.cambridge.**CambridgeSetup <init>: Establishing >> >>> cambridge view resolver >> >>> I 2012-09-12 10:56:21.792 >> >>> st.voo.tick.GuiceConfig contextInitialized: Guice initialization took >> 514 millis >> >>> W 2012-09-12 10:56:21.888 >> >>> Failed startup of context >> >>> com.google.apphosting.utils.**jetty.**RuntimeAppEngineWebAppContext@* >> *1479784{/,/base/data/home/**apps/s~voost0/2012-09-10-1715.**361669184733923098} >> >> >>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to >> >>> instantiate MessageBodyReader >> >>> at org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** >> register(RegisterBuiltin.java:**35) >> >>> at >> >>> org.jboss.resteasy.spi.**ResteasyDeployment.start(**ResteasyDeployment.java:211) >> >> >>> at org.jboss.resteasy.plugins.**server.servlet.** >> ServletContainerDispatcher.**init(**ServletContainerDispatcher.**java:67) >> >> >>> at org.jboss.resteasy.plugins.**server.servlet.** >> FilterDispatcher.init(**FilterDispatcher.java:39) >> >>> at st.voo.tick.util.**GuiceResteasyFilterDispatcher.**init(** >> GuiceResteasyFilterDispatcher.**java:48) >> >>> at >> >>> com.google.inject.servlet.**FilterDefinition.init(**FilterDefinition.java:114) >> >> >>> at com.google.inject.servlet.**ManagedFilterPipeline.** >> initPipeline(**ManagedFilterPipeline.java:98) >> >>> at com.google.inject.servlet.**GuiceFilter.init(GuiceFilter.* >> *java:172) >> >>> at >> >>> org.mortbay.jetty.servlet.**FilterHolder.doStart(**FilterHolder.java:97) >> >> >>> at >> >>> org.mortbay.component.**AbstractLifeCycle.start(**AbstractLifeCycle.java:50) >> >> >>> at >> >>> org.mortbay.jetty.servlet.**ServletHandler.initialize(**ServletHandler.java:662) >> >> >>> at org.mortbay.jetty.servlet.**Context.startContext(Context.* >> *java:140) >> >>> at >> >>> org.mortbay.jetty.webapp.**WebAppContext.startContext(**WebAppContext.java:1250) >> >> >>> at >> >>> org.mortbay.jetty.handler.**ContextHandler.doStart(**ContextHandler.java:517) >> >> >>> at >> >>> org.mortbay.jetty.webapp.**WebAppContext.doStart(**WebAppContext.java:467) >> >> >>> at >> >>> org.mortbay.component.**AbstractLifeCycle.start(**AbstractLifeCycle.java:50) >> >> >>> at com.google.apphosting.runtime.** >> jetty.AppVersionHandlerMap.**createHandler(**AppVersionHandlerMap.java:219) >> >> >>> at com.google.apphosting.runtime.** >> jetty.AppVersionHandlerMap.**getHandler(**AppVersionHandlerMap.java:194) >> >>> at com.google.apphosting.runtime.**jetty.** >> JettyServletEngineAdapter.**serviceRequest(**JettyServletEngineAdapter.**java:134) >> >> >>> at com.google.apphosting.runtime.** >> JavaRuntime$RequestRunnable.**run(JavaRuntime.java:447) >> >>> at com.google.tracing.**TraceContext$**TraceContextRunnable.* >> *runInContext(TraceContext.**java:452) >> >>> at com.google.tracing.**TraceContext$** >> TraceContextRunnable$1.run(**TraceContext.java:459) >> >>> at >> >>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >> >> >>> at com.google.tracing.**TraceContext$** >> AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >> >> >>> at com.google.tracing.**TraceContext$** >> AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >> >> >>> at com.google.tracing.**TraceContext$** >> TraceContextRunnable.run(**TraceContext.java:456) >> >>> at com.google.apphosting.runtime.** >> ThreadGroupPool$PoolEntry.run(**ThreadGroupPool.java:251) >> >>> at java.lang.Thread.run(Thread.**java:679) >> >>> Caused by: java.lang.RuntimeException: Unable to instantiate >> MessageBodyReader >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** >> registerProvider(**ResteasyProviderFactory.java:**761) >> >>> at org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** >> registerProviders(**RegisterBuiltin.java:70) >> >>> at org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** >> register(RegisterBuiltin.java:**31) >> >>> ... 27 more >> >>> Caused by: java.lang.SecurityException: Unable to get members for >> >>> class org.jboss.resteasy.plugins.**providers.DataSourceProvider >> >>> at >> >>> com.google.appengine.runtime.**Request.process-**b6ca2b194d66ed23(Request.java) >> >> >>> at java.lang.Class.**getConstructors(Class.java:**291) >> >>> at org.jboss.resteasy.util.**PickConstructor.** >> pickSingletonConstructor(**PickConstructor.java:27) >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** >> getProviderInstance(**ResteasyProviderFactory.java:**1032) >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** >> addMessageBodyReader(**ResteasyProviderFactory.java:**478) >> >>> at org.jboss.resteasy.spi.**ResteasyProviderFactory.** >> registerProvider(**ResteasyProviderFactory.java:**757) >> >>> at org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** >> registerProviders(**RegisterBuiltin.java:70) >> >>> at org.jboss.resteasy.plugins.**providers.RegisterBuiltin.** >> register(RegisterBuiltin.java:**31) >> >>> at >> >>> org.jboss.resteasy.spi.**ResteasyDeployment.start(**ResteasyDeployment.java:211) >> >> >>> at org.jboss.resteasy.plugins.**server.servlet.** >> ServletContainerDispatcher.**init(**ServletContainerDispatcher.**java:67) >> >> >>> at org.jboss.resteasy.plugins.**server.servlet.** >> FilterDispatcher.init(**FilterDispatcher.java:39) >> >>> at st.voo.tick.util.**GuiceResteasyFilterDispatcher.**init(** >> GuiceResteasyFilterDispatcher.**java:48) >> >>> at >> >>> com.google.inject.servlet.**FilterDefinition.init(**FilterDefinition.java:114) >> >> >>> at com.google.inject.servlet.**ManagedFilterPipeline.** >> initPipeline(**ManagedFilterPipeline.java:98) >> >>> at com.google.inject.servlet.**GuiceFilter.init(GuiceFilter.* >> *java:172) >> >>> at >> >>> org.mortbay.jetty.servlet.**FilterHolder.doStart(**FilterHolder.java:97) >> >> >>> at >> >>> org.mortbay.component.**AbstractLifeCycle.start(**AbstractLifeCycle.java:50) >> >> >>> at >> >>> org.mortbay.jetty.servlet.**ServletHandler.initialize(**ServletHandler.java:662) >> >> >>> at org.mortbay.jetty.servlet.**Context.startContext(Context.* >> *java:140) >> >>> at >> >>> org.mortbay.jetty.webapp.**WebAppContext.startContext(**WebAppContext.java:1250) >> >> >>> at >> >>> org.mortbay.jetty.handler.**ContextHandler.doStart(**ContextHandler.java:517) >> >> >>> at >> >>> org.mortbay.jetty.webapp.**WebAppContext.doStart(**WebAppContext.java:467) >> >> >>> at >> >>> org.mortbay.component.**AbstractLifeCycle.start(**AbstractLifeCycle.java:50) >> >> >>> at com.google.tracing.**TraceContext$**TraceContextRunnable.* >> *runInContext(TraceContext.**java:452) >> >>> at com.google.tracing.**TraceContext$** >> TraceContextRunnable$1.run(**TraceContext.java:459) >> >>> at >> >>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >> >> >>> at com.google.tracing.**TraceContext$** >> AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >> >> >>> at com.google.tracing.**TraceContext$** >> AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >> >> >>> at com.google.tracing.**TraceContext$** >> TraceContextRunnable.run(**TraceContext.java:456) >> >>> ... 1 more >> >>> Caused by: java.lang.reflect.**InvocationTargetException >> >>> at sun.reflect.**GeneratedMethodAccessor11.**invoke(Unknown >> Source) >> >>> at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(** >> DelegatingMethodAccessorImpl.**java:43) >> >>> at java.lang.reflect.Method.**invoke(Method.java:616) >> >>> ... 29 more >> >>> Caused by: java.lang.**NoClassDefFoundError: >> java/io/FileOutputStream >> >>> at java.lang.Class.**getDeclaredConstructors0(**Native >> Method) >> >>> at >> >>> java.lang.Class.**privateGetDeclaredConstructors**(Class.java:2406) >> >> >>> ... 29 more >> >>> Caused by: java.lang.**ClassNotFoundException: >> java.io.FileOutputStream >> >>> ... 29 more >> >>> C 2012-09-12 10:56:21.889 >> >>> Uncaught exception from servlet >> >>> javax.servlet.**UnavailableException: Initialization failed. >> >>> at com.google.apphosting.runtime.** >> jetty.AppVersionHandlerMap.**createHandler(**AppVersionHandlerMap.java:228) >> >> >>> at com.google.apphosting.runtime.** >> jetty.AppVersionHandlerMap.**getHandler(**AppVersionHandlerMap.java:194) >> >>> at com.google.apphosting.runtime.**jetty.** >> JettyServletEngineAdapter.**serviceRequest(**JettyServletEngineAdapter.**java:134) >> >> >>> at com.google.apphosting.runtime.** >> JavaRuntime$RequestRunnable.**run(JavaRuntime.java:447) >> >>> at com.google.tracing.**TraceContext$**TraceContextRunnable.* >> *runInContext(TraceContext.**java:452) >> >>> at com.google.tracing.**TraceContext$** >> TraceContextRunnable$1.run(**TraceContext.java:459) >> >>> at >> >>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >> >> >>> at com.google.tracing.**TraceContext$** >> AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >> >> >>> at com.google.tracing.**TraceContext$** >> AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >> >> >>> at com.google.tracing.**TraceContext$** >> TraceContextRunnable.run(**TraceContext.java:456) >> >>> at com.google.apphosting.runtime.** >> ThreadGroupPool$PoolEntry.run(**ThreadGroupPool.java:251) >> >>> at java.lang.Thread.run(Thread.**java:679) >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/EJUrxhiFMp4J. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
