We are back up and running now after 2 hrs of downtime. To whoever fixed it: THANK YOU!!!
To whoever broke it in the first place: SPANKINGS!!! Jeff On Wed, Sep 12, 2012 at 12:11 PM, Jeff Schnitzer <[email protected]> wrote: > More information: > > * The failure began at 10:54am (pacific). > * Same app on different appid has the same problem. > > (as listed in the stacktraces, the appid is voost0) > > Jeff > > On Wed, Sep 12, 2012 at 11:53 AM, Jeff Schnitzer <[email protected]> wrote: >> HEEEEEEEEEEEEELP! >> >> We have tried everything at this point. Shut down instances, tried to >> deploy a new version, even tried old versions. We've reported a >> production issue. Something is broken inside of GAE. The Guice error >> must be a symptom; the smoking gun seems to be: >> >> java.lang.ClassNotFoundException: java.io.FileOutputStream >> >> We've been down for 30 mins and getting complaints from our clients. >> This looks really, really bad. It's my nightmare scenario - an outage >> in GAE that is small enough not to raise major alarm bells, but >> nevertheless cripples my business. It is not the first time this has >> happened. It is shaking my faith in GAE. >> >> Jeff >> >> On Wed, Sep 12, 2012 at 11:11 AM, Jeff Schnitzer <[email protected]> wrote: >>> Our app has been running fine on the same version, no code deploys >>> (our versions are timestamped so this is 100% certain), for two days. >>> All of a sudden (10 mins ago) our app stopped running. Every attempt >>> to run a request produces this cryptic message: >>> >>> -------- >>> >>> 2012-09-12 10:56:21.791 >>> com.google.inject.servlet.GuiceFilter setPipeline: Multiple Servlet >>> injectors detected. This is a warning indicating that you have more >>> than one GuiceFilter running in your web application. If this is >>> deliberate, you may safely ignore this message. If this is NOT >>> deliberate however, your application may not work as expected. >>> D 2012-09-12 10:56:21.792 >>> st.voo.tick.util.cambridge.CambridgeSetup <init>: Establishing >>> cambridge view resolver >>> I 2012-09-12 10:56:21.792 >>> st.voo.tick.GuiceConfig contextInitialized: Guice initialization took 514 >>> millis >>> W 2012-09-12 10:56:21.888 >>> Failed startup of context >>> com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@1479784{/,/base/data/home/apps/s~voost0/2012-09-10-1715.361669184733923098} >>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to >>> instantiate MessageBodyReader >>> at >>> org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:35) >>> at >>> org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:211) >>> at >>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:67) >>> at >>> org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.init(FilterDispatcher.java:39) >>> at >>> st.voo.tick.util.GuiceResteasyFilterDispatcher.init(GuiceResteasyFilterDispatcher.java:48) >>> at >>> com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:114) >>> at >>> com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:98) >>> at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:172) >>> at >>> org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97) >>> at >>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) >>> at >>> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:662) >>> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140) >>> at >>> org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250) >>> at >>> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) >>> at >>> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467) >>> at >>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) >>> at >>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219) >>> at >>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194) >>> at >>> com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134) >>> at >>> com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) >>> at >>> com.google.tracing.TraceContext.runInContext(TraceContext.java:701) >>> at >>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) >>> at >>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) >>> at >>> com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) >>> at java.lang.Thread.run(Thread.java:679) >>> Caused by: java.lang.RuntimeException: Unable to instantiate >>> MessageBodyReader >>> at >>> org.jboss.resteasy.spi.ResteasyProviderFactory.registerProvider(ResteasyProviderFactory.java:761) >>> at >>> org.jboss.resteasy.plugins.providers.RegisterBuiltin.registerProviders(RegisterBuiltin.java:70) >>> at >>> org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:31) >>> ... 27 more >>> Caused by: java.lang.SecurityException: Unable to get members for >>> class org.jboss.resteasy.plugins.providers.DataSourceProvider >>> at >>> com.google.appengine.runtime.Request.process-b6ca2b194d66ed23(Request.java) >>> at java.lang.Class.getConstructors(Class.java:291) >>> at >>> org.jboss.resteasy.util.PickConstructor.pickSingletonConstructor(PickConstructor.java:27) >>> at >>> org.jboss.resteasy.spi.ResteasyProviderFactory.getProviderInstance(ResteasyProviderFactory.java:1032) >>> at >>> org.jboss.resteasy.spi.ResteasyProviderFactory.addMessageBodyReader(ResteasyProviderFactory.java:478) >>> at >>> org.jboss.resteasy.spi.ResteasyProviderFactory.registerProvider(ResteasyProviderFactory.java:757) >>> at >>> org.jboss.resteasy.plugins.providers.RegisterBuiltin.registerProviders(RegisterBuiltin.java:70) >>> at >>> org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:31) >>> at >>> org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:211) >>> at >>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:67) >>> at >>> org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.init(FilterDispatcher.java:39) >>> at >>> st.voo.tick.util.GuiceResteasyFilterDispatcher.init(GuiceResteasyFilterDispatcher.java:48) >>> at >>> com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:114) >>> at >>> com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:98) >>> at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:172) >>> at >>> org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97) >>> at >>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) >>> at >>> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:662) >>> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140) >>> at >>> org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250) >>> at >>> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) >>> at >>> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467) >>> at >>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) >>> at >>> com.google.tracing.TraceContext.runInContext(TraceContext.java:701) >>> at >>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) >>> at >>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) >>> ... 1 more >>> Caused by: java.lang.reflect.InvocationTargetException >>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:616) >>> ... 29 more >>> Caused by: java.lang.NoClassDefFoundError: java/io/FileOutputStream >>> at java.lang.Class.getDeclaredConstructors0(Native Method) >>> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406) >>> ... 29 more >>> Caused by: java.lang.ClassNotFoundException: java.io.FileOutputStream >>> ... 29 more >>> C 2012-09-12 10:56:21.889 >>> Uncaught exception from servlet >>> javax.servlet.UnavailableException: Initialization failed. >>> at >>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:228) >>> at >>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194) >>> at >>> com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134) >>> at >>> com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) >>> at >>> com.google.tracing.TraceContext.runInContext(TraceContext.java:701) >>> at >>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) >>> at >>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) >>> at >>> com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) >>> at >>> com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) >>> at java.lang.Thread.run(Thread.java:679) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
